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Preface 


“What I want to talk about is the problem of manipulating and controlling things 
on a small scale” stated Richard P. Feynman at the beginning of his visionary talk 
“There’s Plenty of Room at the Bottom”, given on December 29th 1959 at the 
annual meeting of the American Physical Society at the California Institute of 
Technology. Today, almost half a century after this first insight into unlimited 
opportunities on the nanoscale level, we still want — and have to — talk about the 
same issue. The problem identified by Feynmann turned out to be a very difficult 
one due to a lack of understanding of the underlying phenomena in the nanoworld 
and a lack of suitable nanohandling methods. This book addresses the second issue 
and tries to contribute to the tremendous effort of the research community in 
seeking proper solutions in this field. 

Automated robot-based nanomanipulation is one of the key challenges of 
microsystem technology and nanotechnolgy, which has recently been addressed by 
a rising number of R&D groups and companies all over the world. Controlled, 
reproducible assembly processes on the nanoscale will enable high-throughput 
manufacturing of revolutionary products and open up new application fields. The 
ultimate goal of these research activities is the development of automated 
nanomanipulation processes to build a bridge between existing precise handling 
strategies for micro- and nanoscale objects and aspired high-throughput fabrication 
of micro- and nanosystems. These activities include, amongst others, the deve- 
lopment of new nanohandling robots; the investigation of application-specific 
nanohandling strategies; the construction of new application-specific tools; the 
development of advanced control approaches; as well as the investigation of 
suitable sensing technologies. Real-time sensory feedback and fast and precise 
control systems are of particular importance for automated nanohandling, so the 
book will take a thorough look at these issues. 

Despite the growing interest in automated nanomanipulation, there is hardly 
any publication that treats this research in a coherent and comprehensive way. This 
book is an attempt to provide the researcher with an overview of some important 
aspects of this rapidly expanding technology. The other main purpose of this book 
is to inform the practicing engineer and the engineering student about automation 
on the nanoscale as well as the promising fields of application. The latter can be of 
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a very different nature as nanohandling is strongly interdisciplinary in character so 
that the borders between established scientific and technical disciplines fade. 

The idea of the book originates from the lecture courses on microrobotics and 
nanohandling which have been given to students of computer sciences and physics 
at the University of Oldenburg since 2001. At the same time, the book is a 
comprehensive summary of research work that has been performed by my teams at 
the Division of Microrobotics and Control Engineering at the same university as 
well as at the Division of Microsystems Technology and Nanohandling at the 
Oldenburg Institute for Information Technology (OFFIS) for the last six years. All 
the contributors are — or were for a long time — members of the Divisions’ research 
staff. 

It is obviously impossible to pick up every idea and every piece of research 
work on nanohandling and automation on the nanoscale that has been discussed in 
the literature. A representative selection of them was made in the overview section 
of each chapter, and the authors believe that most relevant results have been 
covered. Many of the nanohandling approaches and devices presented in the book 
are at the forefront of technology. Eventually, they will reach maturity and open up 
a mega-market for nanotechnology products. The market penetration and success 
will be caused to a great extent by the innovators who are currently experimenting 
with automated handling on the nanoscale. It is the strong wish of the authors' 
team that this work will help to generate an awareness of this new, diversified 
technology and to guide the interested reader. 

This work was done by the team of researchers involved in quite a few 
international and German joint research projects. Any active researcher would 
understand how difficult it is to spare the time for serving the research community 
by writing a book. For this reason, my strongest vote of thanks goes to all the 
authors who have contributed to this book. I especially want to thank Professor 
Duc Truong Pham, the Director of the Manufacturing Engineering Centre at 
Cardiff University and the scientific editor of the Springer book series on 
Advanced Manufacturing, for triggering the idea of writing a book about my field 
of research. The linguistic proofreading was done by Nicholas Pinfield and 
Christian Fatikow. We are indebted to them for many suggestions that have 
improved the book a great deal. We appreciate the support by Professor Sylvain 
Martel, the Director of the NanoRobotics Lab at Montreal University, who read 
the manuscript and made a lot of valuable comments. We are grateful to the 
colleagues who provided us with graphs and pictures which make it much easier to 
understand the text. The book team had much help with the time-consuming 
drawing of the artwork: we are indebted to Sascha Fatikow for his excellent work. 
Dr. Markus Kemper deserves our sincere thanks for his time and effort with the 
meticulous preparation of the final manuscript for printing. Our thanks also go to 
Daniel Jasper and Dr. Kwangsoo Kim, who helped us with error checking and 
correction in the final manuscript. 


Oldenburg, March 2007 


Sergej Fatikow 
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Trends in Nanohandling 


Sergej Fatikow 


Division of Microrobotics and Control Engineering, 
Department of Computing Science, 
University of Oldenburg, Germany 


1.1 Introduction 


The handling of micro- and nanoscale objects is an important current trend in 
robotics. It is often referred to as nanohandling, having in mind the range of 
aspired positioning accuracy. The Greek word “nanos” (dwarf) refers to the phy- 
sical unit of a nanometer = 1 nm = 10°m. In this book, we understand nano- 
handling as the manipulation of microscale and nanoscale objects of different 
nature with an accuracy in the (sub-) nanometer range, which may include their 
finding, grasping, moving, tracking, releasing, positioning, pushing, pulling, 
cutting, bending, twisting, etc. Additionally, different characterization methods, 
e.g., indenting or scratching on the nanoscale, measurement of different features of 
the object, requiring probe positioning with nanometer accuracy, structuring or 
shaping of nanostructures, and generally all kinds of changes to matter at nanolevel 
could also be defined as nanohandling in the broadest sense. 

Obviously, not all conceivable nanohandling operations are based on robotics, 
e.g., the so-called self-assembly, which will be introduced later. This book does not 
attempt to cover the whole palette of nanohandling options and will confine itself 
to the approaches that can be implemented and eventually automated with the help 
of microrobots with nanohandling capabilities. As in the field of industrial 
robotics, where humans leave hard, unacceptable work to robots, microrobots can 
help humans to handle extremely small objects with very high accuracy. 
Drastically miniaturized robots, or microrobots, are able to operate in extremely 
constricted work spaces, e.g., under a light microscope or in the vacuum chamber 
of a scanning electron microscope (SEM). In particular, microsystem technology 
(MST) and nanotechnology require this kind of robot, since humans lack cap- 
abilities in manipulation at those scales. Automated nanohandling by microrobots 
will have a great impact in both these technologies in the near future and will 
contribute to the development of high-throughput nanomanipulation processes. 
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Microsystem technology aims at producing miniature systems involving micro- 
machined components or structures. Such systems enable new functions and new 
applications as well as having cost benefits. [1] provides a good overview of the 
present and future applications of MST. The emphasis of microsystem technology 
is clearly on the system aspect. However, the vast majority of today’s MST 
products are components, which have to be further integrated into complete micro- 
systems. This integration often requires the robot-based handling of microscale 
objects with an accuracy in the nanometer range. 

Nanotechnology is a new approach that refers to understanding and mastering 
the properties of matter at the nanoscale as well as to the miniaturization of devices 
down to the nanometer level. At this level, matter exhibits different and often 
amazing properties, which leads to a revolutionary potential in terms of possible 
impact on industrial production routes. Nanotechnology offers possible solutions to 
many current problems by means of smaller, lighter, faster, and better performing 
materials, components, and systems. A good overview of the most promising appli- 
cations of nanotechnology is given in [2, 3]. Nanotechnology is now just at the 
beginning of its commercial development. There are still many challenges to be 
solved, in order to be able to control single atoms, molecules, or nanoscale par- 
ticles in the best possible way. The ultimate goal of nanohandling here is nano- 
assembly — the organization of nanoscale objects into ordered structures — with 
precise control over the individual objects’ relative positions. 

The transfer of classical robotic “know-how” from our macroscopic world to 
the world of microscale and nanoscale objects being handled is, however, a huge 
technological challenge for the robotics research community. Some critical issues, 
e.g., parasitic surface forces derogating the positioning accuracy, are regarded in 
Chapter 2. New, advanced actuator and sensor technologies which are suitable for 
nanohandling have to be investigated. Another crucial issue is the development of 
control architectures and methods tailored to the demands of automated nano- 
manipulation. The state of the art for nanohandling control approaches includes 
teleoperated and semi-automated control strategies. The reader will find a good 
review of current work on these approaches in [4]. Here, the operator controls the 
nanohandling robot directly or sends task commands to the robot controller, using 
vision, force or tactile feedback to control the nanohandling process. However, it is 
usually rather slow and not repeatable. In the automatic control approach, the robot 
has closed-loop control using sensory information without any user intervention. 
The latter approach is very challenging, especially due to the difficulty in getting 
available real-time nanoscale visual feedback and the lack of advanced control 
strategies able to deal with changing and uncertain physical parameters and 
disturbances [5, 6]. This book tries to show some solutions to these problems and 
to present promising applications, where tremendous benefit can be gained from 
the controlled handling of matter on the nanoscale and where smart microrobots 
may play an important role both as a high-throughput automated nanohandling 
technology as well as a complementary process to other techniques. 
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1.2 Trends in Nanohandling 


There are several ways to classify nanohandling approaches. The following three 
approaches are being pursued by the majority of the nanohandling research groups, 
and they seem to be most promising and versatile for future developments in this 
field: 


e top-down approach utilizing serial nanohandling by microrobot systems. 
The main goal is the miniaturization of robots, manipulators, and their tools 
as well as the adaptation of the robotic technology (sensing, handling, con- 
trol, automation, efc.) to the demands of MST and nanotechnology. This 
approach is the topic of this book and will be shortly introduced in Section 
1.3. 

e bottom-up approach or self-assembly utilizing parallel nanohandling by 
autonomous organization of micro- and nanoobjects into patterns or struc- 
tures without human intervention. 

e the use of a scanning probe microscope (SPM) as a nanohandling robot. 
In this approach, the (functionalized) tip of an atomic force microscope 
(AFM) probe or of a scanning tunneling microscope (STM) probe acts as a 
nanohandling tool affecting the position or the shape of a nanoscale part. 
Actually, SPM-based nanohandling has to be pigeonholed into the top- 
down drawer, regarding a microscope scanner as a nanomanipulator and 
microscope probe as a nanohandling tool. However, it is worth taking a 
more precise look at this fascinating technology. 


Several other approaches, e.g., the use of optical tweezers [7—10] or electrophoresis 
[11, 12] might also be adapted for automated nanohandling. They are primarily 
used for the manipulation of fragile biological samples because of the low grasping 
forces lying in the pN range. The latter is clearly one of the limitations of these 
non-contact manipulators. 


1.2.1 Self-assembly 


Self-assembly can be considered as a new strategy for parallel nano- and micro- 
fabrication, which draws its inspiration from numerous examples in nature: self- 
assembly is one of the most important strategies used by nature for the deve- 
lopment of complex, functional structures. Recent technological advances have led 
to the development of many novel “bottom-up” self-assembly strategies capable of 
creating ordered structures with a wide variety of tunable properties. In this con- 
text, self-assembly can be defined as the spontaneous formation of higher ordered 
structures from basic units. This approach is increasingly being exploited to 
assemble systems at the micro- and nanoscale. Especially at the nanoscale and 
when the assembly process deals with a large number of parts, the ability to 
efficiently manipulate single parts gradually diminishes, as the size of objects 
decreases, and the need for a parallel manipulation method arises. 

Recent research activities on self-assembly in the microscale and nanoscale 
have been reviewed in [13, 14]. Generally, the self-assembly process involves 
recognition and making connections to the other parts of the system. For this 
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reason, each part has to be equipped with a mechanism supporting its process of 
self-assembly, i.e., the ability to recognize (self-assembly programming me- 
chanism) and connect (self-assembly binding/driving force) to the proper adjacent 
part or template. Additionally, an external agitation mechanism is often needed to 
drive the system to the global energy minimum that corresponds to the correct self- 
assembly. 

The goal of self-assembly in the microscale is usually the exact planar posi- 
tioning of parts onto a substrate (2D self-assembly) or the creation of 3D micro- 
structures which cannot be fabricated by existing micromachining methods. Self- 
assembly is enabled by pre-programming the structure and the functions of parts 
during their synthesis so that parts self-assemble into ordered 2D and 3D archi- 
tectures under appropriate conditions [15, 16]. Typically, a large number of parts to 
be self-assembled are put into a fluid on the substrate surface. The parts in the fluid 
flow are “looking” for the suitable binding sites and spontaneously build an 
ordered structure on the substrate. To guide the self-assembly, e.g., gravitational, 
magnetic, or capillary forces can be utilized. A typical application of gravity 
includes the agitation of parts to make them move on the substrate surface until 
they “find” suitable binding sites — particularly shaped recesses in the substrate — 
and get stuck in them [17, 18]. Capillary forces are increasingly used to guide 2D 
self-assembly [19-21]. Usually, the self-assembly is performed by exploitation of 
the hydrophobic and/or hydrophilic features of substrate and microparts, which can 
be modulated in different ways to improve controllability and selectivity. Electro- 
static forces can also be utilized to build ordered planar microstructures [22-24]. 
The main advantage of this approach is the ability to dynamically control the self- 
assembly process by modulating the electric force. In comparison to 2D self- 
assembly, the 3D approaches are just at the very beginning of their active inve- 
stigation. Again, a liquid is usually used as the medium for guided self-assembly 
and several promising approaches have been demonstrated [25-27]. 

In the nanoscale, self-assembly may enable many of the most difficult steps in 
nanofabrication, including those involving atomic-level modifications of structure. 
As a result, ordered structures with sub-nanometer precision can be expected, both 
in 2D and 3D architectures. Typically, the self-assembly of nanoobjects, e.g., 
nanocrystals, nanowires, or carbon nanotubes (CNT), is triggered using chemistry 
and exploits biologically inspired interaction paradigms such as shape comple- 
mentarity, van der Waals forces, hydrogen bonding, hydrophobic interactions, or 
electrostatic forces. Maybe the best-known example of self-assembled nano- 
structures is the so-called self-assembled monolayers (SAM) which are built from 
organic molecules that chemically bind to a substrate and form an ordered lattice 
[28-30]. SAM can be used for the modulation of surface-dependent phenomena 
(30, 31], which is of interest for different applications of nanotechnology, 
especially for nanoelectronics and nanooptics. Also, 3D self-assembled nano- 
structures are possible [32, 33], e.g., utilizing a molecular recognition process for 
binding complementary DNA strands. This approach also enables part-to-part and 
part-to-substrate self-assembly by using DNA hybridization [34-35], which is a 
highly selective programmable process for generating 3D structures with nanoscale 
precision. Self-assembly of nanowires and CNT has recently attracted significant 
attention. The reason is to pursue many promising applications both in nano- 
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electronics and nanooptics as well as in nano—micro interface technologies. The 
assembly of nanowires and CNT is a challenging task due to their shape 
anisotropy, which makes their proper integration into a device difficult [13, 14]. 
Electric fields between the electrodes on a substrate are widely used in dealing 
with this task and to trigger the self-alignment of rod-shaped nanoobjects [11, 36- 
38]. The above-mentioned SAM approach is another option for self-assembling 
CNT, which is based on the fabrication of binding sites through SAM patterning 
[39]. 

It is obvious that self-assembly has the potential to radically change the auto- 
mated fabrication of microscale and nanoscale devices, as it enables the parallel 
handling of very different objects in a very selective and efficient way. However, 
despite promising results achieved up to now, this technology still remains on the 
level of basic research. Critical challenges in the development of future devices 
through self-assembly are the limited availability of suitable integration tools that 
enable automatic, site-specific, localization and integration of parts into the system, 
especially when the number of sites is very large, as well as the increasing 
complexity of parts due to the necessary fabrication steps for the implementation of 
binding features. The study of defects in self-assembled systems and the intro- 
duction of fault-tolerant architectures like in biological systems will also play a 
prominent role in transferring self-assembly from research laboratories to device 
manufacturing [13]. These challenges are currently being addressed by the self- 
assembly research community. 

Usually, robot-based assembly (top-down) and self-assembly (bottom-up) are 
investigated separately. However, hybrid approaches, using the advantages of 
both serial and parallel technologies, seem to be a promising solution that is worth 
pursuing for different applications in order to achieve higher complexity or pro- 
ductivity [40-43]. A major European research project that started in 2006 aims at 
combining ultra-precision robots with innovative self-assembly technologies, with 
the goal of developing a new versatile 3D automated production system with a 
positioning accuracy of at least 100 nm for complex microscale products [43]. The 
combination of serial robot-based assembly and parallel self-assembly has never 
yet been achieved at the industrial scale, and the project team is anxious to prove 
the viability of this new production concept. 

To sum up, self-assembly is a fascinating research field attracting a rapidly 
increasing number of research groups from multiple disciplines. There is a clear 
indication that self-assembly can be exploited as a supporting technology, and it 
will be able to contribute to automated robot-based assembly approaches. The 
ability to make a complete device by only using self-assembly steps and to become 
one of the key assembly approaches for the products of MST and nanotechnology 
remains to be seen. 


1.2.2 SPM as a Nanohandling Robot 


SPMs can deliver high-resolution images of a wide class of hard and soft samples, 
which are used, e.g., for materials and surface sciences, bioscience research, or 
nanotechnology. Additionally, these devices can be used to interact with nanoscale 
parts, which results in a change of their position or their shape. The first nano- 
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manipulation was reported in 1990 by IBM researchers, who “wrote” the IBM logo 
with xenon atoms by nanomanipulation with an STM [44]. It was the beginning of 
the active investigation of this novel nanohandling approach, especially by using 
AFMs, which offer the widest range of applications in the SPM field. 

The nanohandling capabilities of the AFM were discovered rather by accident 
during AFM imaging scans; some particles just could not be found on the sub- 
strate, because they were moved apart by the AFM tip in the previous scan. 
Controllable positioning of nanoscale parts by an AFM acting as nanohandling 
robot has been actively investigated in the last decade [45-50]. The ultimate goal is 
to automatically assemble nanoscale objects in nanosystems in ambient con- 
ditions, aiming at the rapid prototyping for nanodevices. 

To control the movement of a part, it has first to be localized on the substrate 
by an imaging scan performed in dynamic mode. In the second step, the AFM tip is 
brought by the AFM control system to the immediate vicinity of the part and is 
moved afterwards — staying in dynamic mode without AFM feedback in the z- 
direction — to the centres of the part towards a predetermined location. As a result, 
the part is pushed in a “blind” feed-forward way by repulsive forces [45, 48]. The 
precondition is the AFM’s ability to perform one-line scans in any direction on the 
substrate surface. The re-imaging of the area of interest afterwards reveals the 
results of the manipulation, which are often not satisfying and require frequent 
trial-and-error experiments. Current research work aims at developing a high-level 
AFM control system to perform predictable nanohandling operations, which 
might open the door to high-throughput automated nanomanipulation processes 
[47-52]. Also, several other SPM modes have been used for pushing nanoparticles 
or molecules [46, 53, 54]. The whole variety of operational modes of SPM [55] has 
not been fully investigated in regard to nanomanipulation, which often makes trial- 
and-error experiments necessary for a given task. 

Besides the positioning of nanoscale parts, the SPM tip can also be used to 
modify surfaces with nanometer resolution or to change the object shape, e.g., by 
scratching, indenting, cutting, dissecting, etc. [56-62]. A destructive interaction 
between tip and sample is usually an unwanted effect while imaging. However, for 
nanomachining purposes, the SPM tip can be exploited as a nanohandling tool, 
e.g., milling cutter, nanoscalpel, or nanoindenter. Nanoscratching can be imple- 
mented by moving an AFM tip on a surface and applying a high load force on the 
tip. The technique can be used amongst others for mask-free lithography on the 
nanoscale level. Biological specimens can also be handled in this way. The chro- 
mosomal microdissection by AFM can e.g. be used for isolating DNA [63, 64]. 
The AFM is applied first in non-contact mode or in tapping mode for the loca- 
lization of the cut site in the genetic material. After that, a DNA chromosome is 
extracted by one AFM linescan and picked up by the AFM tip through hydrophilic 
attraction. 

“Writing” on a substrate surface by the AFM tip is another interesting option 
for shaping on the nanoscale. The mask-free nanolithography mentioned above can 
be implemented not only by nanoscratching but also by anodic oxidation [65, 66] 
or by the so-called dip-pen lithography (DPN) [67, 68]. A line width of a few tens 
of nanometers can be achieved by both approaches. To perform nanostructuring by 
anodic oxidation, a nanometer-thin metal layer is deposited on the substrate 
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surface, and a voltage is applied between the metal and the conductive AFM tip. 
Since the metal surface is moistened in ambient atmosphere, an electrolytic process 
is triggered by the voltage, resulting in a tiny metal oxide dot on the surface. By 
proper process control, these dots can form a sophisticated nanoscale pattern. 

As the name of the approach implies, dip-pen lithography works in a manner 
analogous to that of a dipped pen. Chemical reagents (“ink”) are transported from 
the AFM tip to nanoscopic regions of a target substrate by using capillary forces. 
This direct-writing process enables the building of nanoscale structures and 
patterns on different surfaces by literally drawing molecules onto a substrate. The 
AFM tip is coated with the ink that is to be deposited, and the molecules of the ink 
are delivered to the surface through a solvent meniscus forming between the tip 
and the substrate under ambient conditions. This simple method of directly depo- 
siting molecules onto a substrate has recently become an attractive tool for nano- 
scientists, especially because of its versatility. The approach enables molecular 
deposition of virtually any material (hard and soft) on any substrate. However, 
ink/substrate combinations must be chosen carefully, so that the ink does not 
agglomerate or diffuse. Additionally, the ink molecules have to be able to anchor 
themselves to their deposition location (molecular “glue”). These challenges are 
the subject of the current basic research activities in this field. 

Going back to nanomanipulation and taking into account the primary concern 
of this book, automated nanohandling, the main drawback of AFM-based nano- 
handling is the lack of real-time visual feedback. The same AFM tip cannot be 
simultaneously used for both imaging and handling, so that the results of nano- 
handling have to be frequently visualized by an AFM scan to verify the perfor- 
mance. This procedure makes the nanohandling process inefficient, rather un- 
suitable for high throughput, and includes uncertainties due to being “blind” during 
the actual manipulation. Several research groups are trying to overcome this 
problem by modeling the nanohandling task, which might enable nanomani- 
pulation in open-loop mode, without visual feedback. Having a valid model of the 
nanopart behavior, including all relevant interactions between tip, part and 
substrate, it might be possible to mathematically simulate the behavior of the 
nanoobjects during manipulation and to calculate the expected position of the part 
in real-time. Such a model is the basis of the so-called augmented reality systems 
“translating” the nanoworld into virtual reality and delivering calculated “visual 
feedback” during manipulation [69, 70]. This approach, however, requires exact 
knowledge of nanomanipulation phenomena, which is not available in the current 
state of the nanosciences. The usability, therefore, of augmented reality systems for 
automated nanomanipulation, especially in regard to reliability and reproducibility, 
is currently limited due to a lack of understanding of what exactly is going on 
during nanomanipulation. 

Another problem in regard to nanohandling automation arises when accuracies 
in the sub-nanometer range are required [71]. Most commercial devices cannot 
offer a reliable position feedback at this level, and spatial uncertainty in AFM — 
because of the thermal drift of AFM components, creep and hysteresis of piezo- 
actuators, and other variant effects and nonlinearities — cannot be taken care of in 
a direct way. Some solutions to this problem have been addressed in [50, 72, 73]. 
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From the automation point of view, a combination of the AFM as a nano- 
handling robot with other imaging techniques that supply independent visual feed- 
back from the work scene during nanomanipulation by the AFM seems to be most 
promising. For positioning accuracies of about 0.5 um and worse, the manipulation 
can be monitored by a light microscope. This approach is frequently used in bio- 
science research to provide multimodal imaging capabilities for yielding extensive 
information on biomolecules and biological processes [74]. 

SPM-SEM hybrid systems are currently attracting rising interest. Here, an 
SPM head is integrated into the vacuum chamber of an SEM, so that both micro- 
scopy methods are used in a complementary fashion to analyze the sample 
properties, building a sophisticated nanocharacterization device [75-77]. To 
exploit such a system for nanohandling, SEM can be used as a sensor for visual 
feedback during nanomanipulation or modification of sample surface by the AFM. 
Important issues to be addressed are the synchronizing of both microscopes and 
proper system engineering enabling the AFM tip to act in the SEM’s field of view. 
The latter usually requires a large tilt of the SPM against the electron beam [76]. 
The use of an environmental SEM (ESEM) may open up new applications as 
vacuum-incompatible samples such as biological cells can also be analyzed or 
handled this way. 

The latter concept already builds a bridge to the topic of this book, which is 
introduced in the next section. Indeed, if we exclude the visualization feature of 
AFM and just think of the nanopositioning capability of the AFM scanner, then we 
are left with a nanopositioning module carrying a tiny cantilever with a (func- 
tionalized) nanoscale tip — a three degrees-of-freedom (DoF) nanohandling robot 
that can be used for diverse applications in the vacuum chamber of the SEM. 


1.3 Automated Microrobot-based Nanohandling 


Different concepts are being followed to carry out micro- and nanomanipulation 
for specific classes of application: 


e Purely manual manipulation is the most often used method today. It is a 
common practice, e.g., in medicine and biological research. Even in 
industry, such tasks are often carried out by specially trained technicians, 
who position the parts with tiny hammers and tweezers under a microscope 
and finally fasten them in the desired position. However, with progressive 
part miniaturization, the tolerances have become smaller and smaller, and 
the capabilities of the human hand are no longer adequate. 

e The application of teleoperated manipulation systems, which transform 
the user’s hand motions into the finer 3D motions of the system manipu- 
lators by a sophisticated man—machine user interface. Here, special effort is 
devoted to the development of methods which allow the transmission of 
feedback information from the work scene (images, forces, noises) in a 
user-friendly form. The user interface can include a haptic device, providing 
tactile information that helps the user to operate in a more intuitive way. 
The haptic device might also be integrated into a virtual reality environment 
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that is based on the mathematical modeling of the application-relevant phe- 
nomena in the nanoworld. However, the fundamental problems of reso- 
lution of the fine motion and of speed as well as of repeatability remain, 
since the motion of the tool is a direct imitation of that of the user’s hand. 

e The use of automated nanohandling desktop stations supported by mini- 
aturized nanohandling robots, which exploit direct drives typically imple- 
mented by using piezoelectric, electrostatic, or thermal microactuators. The 
flexibility of such a microrobot can be enhanced by dividing the actuator 
system into a coarse positioning module (e.g., a mobile platform) and a 
fine positioning module or nanomanipulator carrying an application- 
specific tool. There is no direct connection between the user’s hands and the 
robot. The user commands are given through a graphical user interface to 
the station’s control system, which generates corresponding commands for 
the robot actuators. The degree of abstraction of the user instructions is de- 
termined by the capabilities of the control system. Several microrobots can 
be active at the same time to deal with a handling task. 


Different aspects of the latter approach are discussed in this book, along with 
promising applications in MST, nanotechnology, biotechnology and material 
science. 

Microrobotics for handling microscale and nanoscale parts has been established 
as a self-contained research field for nearly 15 years [78—107]. In recent years, a 
trend towards the microrobot-based automation of nanohandling processes 
emerged, and different concepts are currently being investigated [81, 104, 108- 
127]. Process feedback, i.e., the transmission of information from the nanoworld 
to the macroworld to facilitate the control of the handling process, has emerged as 
the most crucial aspect of nanohandling automation. Vision feedback and force 
feedback are the two information channels to be used for automation purposes. 

With present technology, it is rather difficult to obtain reliable force infor- 
mation, while handling microscale and especially nanoscale parts. Real-time force 
feedback is, on the other hand, the inherent feature of the AFM, so the use of an 
AFM probe for nanohandling may provide the necessary force feedback [80]. As 
optical detection based on a reflected laser beam and utilized in most AFMs 
imposes serious limitations on the robot’s mobility, piezoresistive AFM probes 
seem to be a more practical solution, even though this offers worse resolution 
compared to the laser beam [80]. A few promising approaches are introduced in 
Chapter 6. 

Nevertheless, vision feedback is often the only way to control a nanohandling 
process. The capability of a light microscope rapidly decreases with the parts being 
scaled down to the nanoscale level. Scanning near-field optical microscopy 
(SNOM) may, however, be exploited for nanoscale manipulations in an ambient 
environment [86]. The vacuum chamber of an SEM is for many applications the 
best place for a nanohandling robot. It provides an ample work space, very high 
resolution up to 1 nm, and a large depth of field (see Chapter 2 for more infor- 
mation). Quite a few research groups have recently been investigating different 
aspects of nanohandling in SEM, e.g., [78, 88, 89, 93, 94, 102, 108, 116, 121, 128, 
129]. However, real-time visual feedback from changing work scenes in the SEM 
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containing moving microrobots is a challenging issue, which is thoroughly 
analyzed in Chapter 4. 

Figure 1.1 presents a generic concept of the automated microrobot-based nano- 
handling station (AMNS), first introduced in [130] and further gradually deve- 
loped at the University of Karlsruhe and the University of Oldenburg [120-127]. 
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Figure 1.1. Generic concept of the automated microrobot-based nanohandling station 


Positioning with nanometer precision is the first precondition for the development 
of an AMNS. Typically, the microrobots are driven by piezoactuators with reso- 
lutions down to sub-nm ranges. The travel range is comparatively large with 
several tens of millimeters for stationary microrobots and with almost no limita- 
tion for mobile microrobot platforms [128, 131, 132]. The mobile robots of the 
station have a micromanipulator integrated in their platform, which makes them 
capable both of moving over longer distances and of manipulating with nanometer 
accuracy. The latter leads to more flexibility, as the robots can be deployed every- 
where inside the SEM vacuum chamber. Stationary robots are, on the other hand, 
easier to control, which makes them more suitable for high-throughput automation. 
The robots usually have to be tethered and get power, e.g., driving voltages for the 
robots’ piezoactuators, over the vacuum-sealed mechanical robot interface inte- 
grated into the SEM wall. 

The flexibility of the system can additionally be enhanced by accommodating 
several robots in the station, which can cooperate and carry out handling tasks as 
a team. Moreover, one of the robots can act as a “cameraman”, carrying a minia- 
ture video microscope. Such a mobile microscope can deliver images from 
virtually any point of view in the work space of the nanohandling robots. The 
combination of a mobile video microscope and SEM with an integrated video 
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camera can contribute to the station’s flexibility and versatility, providing a smooth 
transition between the different magnifications during a nanohandling process. 
Various tools can be attached to the manipulator and exchanged according to the 
task to be accomplished. The already-mentioned use of a (functionalized) AFM 
probe with its extremely sharp tip (10—20nm) as a robot tool is currently one of the 
most actively pursued approaches [4, 80, 86, 133]. 

SEM, video cameras, force sensors, as well as — if available — position sensors 
integrated into the robot axes, build the sensor system of the AMNS. SEM 
delivers near-field sensor information for fine positioning of the robot tool, and 
video cameras provide necessary far-field information for the coarse positioning of 
the robot. The sensor data are sent to the station’s control system for real-time 
signal processing. Its task is to calculate the positions of the robots and their tools 
as well as the positions of the parts to be handled or other objects of interest. The 
calculated positions serve as input data for the closed-loop robot control. 

Even though the AMNS is designed for automated nanohandling, teleoperated 
work is also possible. The latter is performed by using a haptic interface and/or a 
graphical user interface (GUI). Teleoperation is often the first step on the way to 
automation, as it helps the user to learn more about the nanohandling task to be 
implemented. A good overview of teleoperation techniques and applications is 
given in [134]. 

The positioning accuracy of the microrobots during automated nanohandling is 
affected by several factors, so that a powerful position control approach is required. 
The latter is run on the low level of the control system. The demands on the 
low-level control system and the implementation results are discussed in detail in 
Chapter 3. The high-level control system is responsible amongst others for path 
planning, error handling, and the parallel execution of tasks. Both user interfaces, 
GUI and haptic interface, are supported by the high-level control system as well. 
An advanced control system architecture that is currently being implemented [135] 
and tailored for nanohandling automation in the SEM is introduced in Section 7.6. 

The AMNS concept has been implemented [136-140] or is currently being 
implemented [141-145] in different application fields in which (semi-) automated 
nanomanipulation is required. The implementation results are presented and on- 
going work is discussed in Chapter 2 and Chapters 6—10. 


1.4 Structure of the Book 


The following is a brief summary of the topics covered in the following chapters. 
Chapter 2 introduces fundamental considerations for the application of a scan- 
ning electron microscope as position sensor of an automated microrobot-based 
nanohandling station. Due to its resolution, its image acquisition times and scal- 
ability, the SEM is the preferred sensor for nanohandling and nanoassembly pro- 
cesses. In order to successfully automate such processes, parasitic forces have to be 
taken care of. Furthermore, the automatic detection of contact between objects or 
their height difference is a major problem. Possible solutions to these problems are 
discussed. Nanohandling and nanoassembly processes are described as a combi- 
nation of simple tasks and subtasks (primitives). Based on this representation, 
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approaches for the optimization of reliability and throughput of the process can be 
defined. One of the first implementations of the AMNS concept, a two-robot 
station for the (semi-) automated handling of silicon lamellae is presented. 

Chapter 3 deals with the closed-loop pose controller for the mobile micro- 
robots of the AMNS. It contains a trajectory controller, a motion controller, and an 
actuator controller. The actuator controller is implemented as a Self-Organizing 
Locally Interpolating Map (SOLIM), resulting in a learning direct inverse model 
controller based on Self-Organizing Maps (SOMs). The performance of the 
SOLIM approach is validated by the learning of an inverse model of a virtual two- 
joint serial kinematic link. Finally, both an automatically and a manually trained 
SOLIM actuator controller are applied to a mobile microrobot platform of the 
AMNS presented in Chapter 2. 

Chapter 4 covers the SEM in combination with real-time image processing 
algorithms, which is proposed as the near-field sensor for the automation of nano- 
handling tasks. The specific properties of an SEM as image sensor are explained, 
and the requirements for real-time image processing methods are outlined. Then, 
the integration of an SEM into an image processing system is demonstrated. The 
latter enables real-time image access and electron beam control. The main part of 
the chapter is devoted to the implementation and validation of real-time tracking 
algorithms in the vacuum chamber of an SEM, using cross-correlation and active 
contours with region-based minimization. 

Chapter 5 deals with a crucial issue of 3D vision feedback for nanohandling in 
an SEM. For precise handling, it is often necessary to know the position of objects 
and tools in all three dimensions of space, i.e., 3D visual information is to be 
acquired. Basic concepts such as stereoscopic imaging in nature are analyzed. In 
addition, approaches for depth detection in the SEM are illustrated, and a 3D 
imaging system, tailored for nanohandling tasks, is presented. The application of 
this system is shown for the handling of two different kinds of nanoobjects. 

Chapter 6 gives an overview of the fundamentals and principles of micro/nano 
force measurement. The main emphasis is placed on the special requirements in 
force feedback for nanohandling by microrobots. Near visual process monitoring 
using force feedback is the most important source of sensor data in nanohandling. 
The main challenge is the measurement of forces in the range of micro- and nano- 
newtons, to control the interaction between parts and tools as well as for nano- 
mechanical characterization of, e.g., nanowires, carbon nanotubes, or biological 
cells. The integration of force feedback provides essential process information for 
both the operator working with telemanipulation devices and the control system 
operating in automatic mode. The state of the art in force microsensors and force 
sensing for robot- and AFM-based nanohandling systems is described. 

Chapter 7 outlines different issues around the characterization and handling of 
carbon nanotubes. The basics of CNTs are mentioned, followed by an analysis of 
structural, electronical, and mechanical properties of CNTs. Fabrication techniques 
and possible applications for CNTs are explained. Characterization techniques are 
outlined, and the advantages of CNT characterization in an SEM are demonstrated. 
The latter requires SEM-tailored nanohandling robot systems; their state of the art 
is discussed. The developed AMNS for the handling and characterization of CNTs 
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is introduced, and preliminary implementation results are shown. Finally, the novel 
control system architecture for automated CNT nanohandling is introduced. 
Chapter 8 deals with the characterization and manipulation of biological objects 
by an atomic force microscope. An overview of relevant parameters for AFM mea- 
surements in liquids and of soft samples is given. After a short introduction of the 
biological background, the current state of the art in AFM-based characterization is 
analyzed, ranging from imaging tasks in biological processes, through conductance 
measurements on DNA, to activation and measurements of stress-activated ion 
channels. The first implementation steps of the AMNS for cell characterization are 
presented, and preliminary results of the experiments with measuring cell elasticity 
are introduced. 

Chapter 9 presents the application of nanohandling microrobotics for nano- 
mechanical characterization. An emphasis is placed on the characterization of very 
thin coatings by using nanoindentation. The first section outlines the theoretical 
background of the well-established method of instrumented indentation for the 
determination of material hardness and Young’s modulus. The second section 
demonstrates the use of an AMNS for the nanoindentation of epoxy-based electri- 
cally conductive adhesives (ECA). The current setup of the station is introduced, 
and the necessary calibration steps for the apparatus are analyzed. Finally, the first 
experimental results of hardness measurements on ECA samples are discussed. 

Chapter 10 covers current research work on Electron Beam-induced Depo- 
sition (EBiD) inside an SEM. EBiD is relevant for the fabrication of nano- 
mechanical elements, e.g., pins, flexible hinges, or more complicated structures, as 
well as for nanoassembly. The latter can be accomplished by EBiD of a suitable 
precursor material between two parts (nanosoldering). Based on the interactions 
between electron beam and substrate, the rate equation model for EBiD is investi- 
gated, and the relevant parameters are analyzed. The molecular flux density has 
been identified as a crucial parameter for the optimization of the growth rate of the 
resulting nanostructures. This parameter can be modulated by using an advanced 
gas injection system (GIS), so design considerations and control methods for 
microrobot-based GIS are discussed. Basic principles of EBiD process control are 
illustrated, and promising methods are introduced. Necessary mechanical data is 
offered from the appropriate literature, and own experimental results are presented. 
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2.1 Introduction 


Within the last ten years, the interest of industry and research and development 
institutes in the handling of micro- and nanometer-sized parts has grown rapidly 
[1]. Micro- and nanohandling has become a very common task in the industrial 
field and in research in the course of ongoing miniaturization. Typical applications 
include the manipulation of biological cells under an optical light microscope, the 
assembly of small gears for miniaturized gearboxes, the handling of lamellae cut 
out of a silicon wafer in the semiconductor industry, and the chemical and physical 
characterization of nanoscale objects. The number of applications for nanohandling 
and nanoassembly is expected to grow rapidly with the development of nano- 
technology. The handling process is the precursor of the assembly process, hence, 
in this chapter, these expressions are used equally where not explicitly stated. 

Often, a distinction is made between macro-, micro-, and nanoscale assembly 
with respect to the part size, where the part dimensions are larger than 1 mm for 
macroscale, smaller than 1 mm for microscale, and smaller than 1 um for nano- 
scale handling [2]. This distinction should even be tightened, because the inter- 
action between the handling system and the handled parts is mostly determined by 
its smallest dimension, which determines the necessary positioning accuracy 
(Chapter 1). A typical example for parts with very exotic aspect ratios, but which 
are still considered as nanometer-sized parts, are nanofibers or nanotubes. They 
can, e.g., be produced by electro-spinning [3], which results in lengths in the cm- 
range. However, most of these handling processes are still accomplished by means 
of manual operation [4-6]. Very often, this leads to either very long process 
durations with high reliability or to shorter durations with low reliability. 

The handling process itself can be distinguished by the number of parts handled 
at a time, i.e., when only one part is handled at a time the expression serial 
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approach is used, in contrast to parallel approach for simultaneous handling of 
multiple parts [2]. These two approaches are based on very different conside- 
rations: the serial approach is the more conservative one, where the principles of 
handling known from the macroscale are adapted to the micro- and nanoscale. 
Naturally, special considerations have to be taken into account when downscaling, 
which is one of the main issues discussed in this chapter. 

The other approach is the parallel handling of micro- and nanometer-sized 
parts, where force fields are used to position and orientate objects. The aim here is 
to maintain the advantages of batch processes, as applied in the MEMS (micro- 
electro-mechanical systems) and semiconductor industries. 

Handling processes can be evaluated in respect of two parameters: throughput 
and reliability. (Massively) parallel handling or manufacturing aims at very high 
throughputs, e.g., assembly of dies in the semiconductor industry. In contrast, the 
serial approach handles only one part at a time, where high reliability is the main 
requirement because of the special value of the handled parts. A typical example is 
the handling of TEM (transmission electron microscope) lamellae that are small 
slices (approx. 20 um x 10 um x 100 nm) cut out of a processed silicon wafer by a 
focused ion beam (FIB). These lamellae are then transferred to a TEM for ins- 
pection, i.e,. the TEM lamellae are the micrographs of the semiconductor industry. 
This approach is very important for discovering failures in semiconductor pro- 
cesses, and high reliability of handling is required. In general, the criteria to be 
considered when distinguishing between a serial and a parallel approach are the 
number of parts to be handled or assembled, the complexity of the process, and the 
individuality of the single parts. 

The given examples for the serial and parallel approach represent two appli- 
cations with very different demands. Still, the goal is always to maximize reli- 
ability and throughput for every handling system, independent of the chosen 
approach, but sometimes reliability is more important than throughput, and vice 
versa. This chapter focuses on automation issues in the field of nanohandling for 
the serial approach. 

The handling of nanoscale objects usually takes place in a special environment 
necessary for observation, e.g,. under optical microscopes or scanning electron 
microscopes (SEM). The advantages and disadvantages of the single vision 
sensors and resulting consequences for the handling of objects with respect to 
automation will be discussed in Section 2.2. When the size of the handled objects 
is reduced, the relationship between surface and volume changes dramatically, 
resulting in a stronger influence of parasitic forces on the objects compared to the 
macroworld. These forces have to be overcome in order to successfully automate 
handling and assembly processes (more information in Section 2.3). Another major 
issue for the process automation discussed in Section 2.3 is the contact detection, 
which is the detection of height distances between objects. Critical issues regarding 
handling processes and the planning of these processes by a combination of simple 
tasks and subtasks will be discussed in Section 2.4. Based on these, measures and 
approaches for optimizing reliability and throughput of handling and assembly 
processes will be described and discussed in Section 2.5. The setup and results 
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achieved with an automated microrobot-based nanohandling station (AMNS), 
implemented for the handling of TEM lamellae, will be described in Section 2.6. 


2.2 Vision Sensors for Nanohandling Automation 


Nanohandling can be seen as the continuation of macrorobotics to the nanometer 
scale, taking several new issues into account. One of these issues is the need for a 
near-field vision sensor (Chapter 1), providing visual information about the 
handling process, i.e., making the nanoworld accessible to the human eye. 

Although nanohandling tasks could be performed without any visualization by 
relying on measurement data from the handling tools like forces, velocity, and 
time, it is much more precise to measure geometric values (position, length, 
distance, efc.) directly using vision-based sensors. As will be shown in Section 2.4, 
especially for nanohandling tasks the continuous gathering of geometric 
information is very important to achieve high reliability. There are three main 
reasons for this: 


1. The parasitic forces result in an apparently unpredictable behavior of 
objects, as many parameters for these parasitic forces are either unknown 
or change continuously. A typical example is the release of a small micro- 
or nanoscale object from a gripper. Opening the gripper jaws does not 
necessarily lead to dropping the object due to gravitation; instead the object 
often sticks to one of the jaws due to parasitic forces. 

2. The sensors that are used in nanohandling and manufacturing to determine 
an object’s state (e.g.,. “gripped”), position (e.g., “distance to the gripper”) 
or orientation (“upright”) are usually bigger in size than the objects them- 
selves. By contrast, when handling parts in the macroscale, the sensors are 
smaller or in the same range as the tools. For example, modern grippers 
can easily handle an egg without damaging it, due to integrated force 
sensors, whereas the integration of force sensors into a gripper with a jaw 
cross-section of a few micrometers is extremely challenging (Chapter 6). 
Thus, on the nanoscale the sensor density, i.e., the amount of sensors per 
handling tool, is much smaller than on the macroscale. 

3. The near-field vision sensors (Chapter 1) considered are global vision 
sensors, i.e., they measure a scene based on a global coordinate system. 


The combination of the first two circumstances — apparently undetermined beha- 
vior of the objects due to hardly determinable parasitic forces and a significantly 
lower sensor density — are the major challenges for any nanohandling process. 

Vision sensors, however, provide a tremendous amount of information, because 
objects can be recognized and their relationship to each other can thus be qualified 
and quantified. A typical example is the gripping of a small glass sphere, where it 
is necessary to know if the sphere is between the gripper jaws. By evaluating this 
information using object recognition, it is possible, e.g., to operate without a force 
sensor on the gripper jaws. 
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2.2.1 Comparison of Vision Sensors for Nanohandling Automation 


Of further interest are the geometric scales, which have to be bridged during 
nanohandling tasks. Consider a typical robot used for assembly tasks in the 
automotive industry. The range of the robot is several meters (~ 10°m), and typical 
position accuracies are in the range of a millimeter (~10° m); the dimension of the 
geometric scale is four orders of magnitude. As a comparison, the range of the 
AMNS described in Section 2.6 is from one decimeter (~10'm) to a position 
accuracy of 100 nm (~107 m), thus seven orders of magnitude are passed through. 
For most applications, it is preferable to have a vision sensor that can be zoomed 
seamlessly, in order to cover the full range of geometric scale. 

The resolution of the imaging sensor is defined by the distance between two 
objects needed to recognize them as separated. Therefore, the resolution indicates 
the size of the object, which the imaging sensor can track in a handling process. 
However, it must be kept in mind that among other factors, the resolution in the 
scanning microscopy is strongly dependent on the image acquisition time, e.g., 
the slower an image is scanned, the better the resolution becomes. 

The image acquisition time has a major influence on the automation process, 
not only with respect to the image quality. It also determines the maximum 
velocity with which objects or tools can be moved under observation. This topic 
will be discussed in Section 2.2.3. 

Of considerable interest for object recognition purposes and for the user is the 
information contained in the image acquired from the sensor. Images based on 
light from the visible spectrum can be colored (light optical microscope) with the 
colors giving information about the geometric surface of the object. By contrast, 
images gathered by an SEM using an energy dispersive X-ray detector (EDX- 
detector) contain information about the material from a depth of up to 3 um from 
the object’s surface. Images gathered by a scanning probe microscope (SPM), e.g., 
an atomic force microscope (AFM), contain information about the tip-sample 
interaction, i.e., the distinction between two objects lying on each other is hardly 
possible without previous knowledge. With regard to automation of handling tasks, 
the necessary information is based on geometric conditions (e.g., “position”, 
“orientation” and “distance”). However, the information contained in an image can 
be of different quality, e.g., material, material contrast, conductivity, or atomic 
forces. Thus the mapping from the image information to the geometric information 
about an object condition can in many cases only be fulfilled with previous 
knowledge. 

The interactions between sensor medium and object have also to be considered 
as the medium influences the object. For example, in the SEM, the electron beam 
used for scanning the object can lead to electrical charging of the object or even to 
damage. The tip of an AFM used for scanning an object can move the object due to 
parasitic forces and thus accidentally interfere with a handling process. 

Very important for handling tasks is the dimensionality of the gathered image. 
Two-dimensional images are common, e.g., in light microscopy or SEM. 
Necessary for handling tasks is often the determination of the geometric condition 
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of the object in three dimensions, which has to be done when only 2D images are 
available. Specialized methods are discussed in Section 2.3.2 and in Chapter 5. 

Other issues are the environmental requirements and the constraints imposed by 
the vision sensor. Typical restrictions are given in installation space, vacuum 
compatibility, and electromagnetic shielding. 

Three typical vision sensors used for nanohandling are given in Table 2.1. The 
light microscope has its major domain as a vision sensor for the handling of 
micro-sized parts, due to its comparatively low resolution. Still, the frame rate is 
only determined by the quality of the camera grabbing the images and not by the 
medium itself, as it is not a scanning vision sensor. Furthermore, the geometric 
information can be directly evaluated. The 2D images with a low depth of focus 
are well suited for automation purposes, as the height difference between two 
objects can easily be quantified (Section 2.3.2). 

Within this book the focus regarding vision sensors is on scanning electron 
microscopes, whose major advantages are their high resolution combined with a 
high range of magnification. The information contained in the images about the 
objects is mapped to geometric conditions, even with the commonly used 
Everhart-Thornley SE detector. Further advantages can be drawn from imaging 
with specialized detectors (e.g., object recognition through material identification). 
A drawback for automation is certainly the comparatively low frame rate. The 
consequences of this issue are considered in Section 2.3, and possible solutions are 
presented in Chapter 4. Additionally, the high depth of focus of the 2D images 
complicates the determination of height distances between objects. Solutions for 
this problem will be discussed in Section 2.3.3. The electron beam scanning the 
objects and tools can also lead to electric charging and thus undetermined parasitic 
forces. 

Atomic force microscopes become more and more interesting for the auto- 
mation of nanohandling tasks where the object’s size is only a few nm. Due to its 
very high resolution, the AFM — or more generally the SPM — is the only option. 
However, its very low frame rate prevents the automation of processes at 
reasonable speeds; this might change when the first high-speed AFMs become 
commercially available [7]. From an automation aspect, the generation of image 
information that can easily be transferred into three-dimensional views of the hand- 
ling process is very advantageous, although a reference level (usually the substrate, 
on which the objects are placed) has to be present. 

Recapitulating the issues discussed above, the conclusion can be drawn that the 
light microscope has the most advantages regarding microhandling. The low 
requirements with regard to the environment and the sensor medium make it a 
comparatively cheap and flexible sensor. Although light microscopes can open the 
door to the nanometer range, automation of nanohandling is hardly possible due to 
a lack of geometric information. For example, silicon nanowires with a diameter of 
a couple of hundred nanometers and a length of several micrometers are visible 
under the light microscope as interference. 


28 Thomas Wich and Helge Hülsen 


The application of AFMs for nanohandling tasks is reasonable for extremely small 
objects. The gap between light microscopy and AFM is best bridged using SEMs. 


Table 2.1. Comparison of light microscope, SEM, and AFM as vision sensors for nanoscale 
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2.2.2 Zoom Steps and Finding of Objects 


Although the handled objects are in the nanometer to micrometer range, they must 
often be transported over several centimeters (e.g., handling of TEM lamellae). For 
assembly applications, micro- to nanometer-sized parts must mostly be joined onto 
parts with a size of several millimeters (e.g., bonding of carbon nanotubes on AFM 
tips). 

Therefore, several orders of magnitude on the geometric scale have to be gone 
through, mostly starting on the centimeter scale and zooming in multiple steps. 
Typical examples for the different zoom steps are given in Figures 2.14 and 2.15. 
Usually, after every zoom step a positioning step follows, in order to center the 
object for the next zoom step. 

Quite often, different actuators and/or sensors are used for different magnifica- 
tions, e.g., a DC motor for coarse positioning at low magnifications and a piezo 
drive for fine positioning at high magnifications. Consequently, every combination 
of a zoom step and its subsequent positioning step has to be seen as a task of its 
own in the process chain with its individual reliability (Section 2.4). Hence, the 
number of zoom-and-center (ZAC) steps should be minimized. It is therefore of 
considerable interest to calculate how many ZAC steps are at least necessary, until 
an object is sufficiently magnified, starting from a defined imaged area. 

For determining the number of steps, the following approach is sensible: the 
imaged area should be square-shaped with an edge length k, ; the minimum edge 


length of the next (k,,,) magnification step can then be determined by the fol- 
lowing equation: 


pi, (2.1) 


Pos n 
Pixel 


k n =H-stu 


In Figure 2.1, the single terms are illustrated. The term H-s represents the 
minimum desired image size, consisting of the structural size s of the object and 
the hull factor H. The structural size s is the size of an object (e.g., diameter of a 
carbon nanotube, CNT) or the distance between object and tool in a handling 
process. The hull factor H determines the image size compared to the object size 
needed for the handling process. 

After every zoom step, the object is centered again. The actuator performing 
this positioning step has an accuracy of up,» which has to be added to the hull. 
Furthermore, the edge length k,,, of the (7+1)th zoom step depends on the 
accuracy of the object recognition of the n-th step, i.e., defining the maximum 
difference between measured and real position. The term Upper’ K, /Ap,., reflects 
the object recognition accuracy, consisting of the number of pixels 4,,.,, per edge 
length k, and the accuracy of the object recognition Uup,e given in pixels. 


ixe. 
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Figure 2.1. Two subsequent zoom-and-center steps during magnification of the object with 
diameter s. The left image has a lower magnification compared to the right one, i.e., 
ka> k,+1. In the right image, the parameters used in Equation 2.2 are shown. The inner 
square is used as the area of movement for the object with a diameter s . 


Based upon the above equation, the edge length of the n-th zoom step can be 


calculated by transferring the recurrence relation in Equation 2.1 into an explicit 
expression for k, : 


k, (6-7 Hs + Up, = p ie H- S + U pos (2.2) 
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Assuming that the edge size of the n-th frame should be equal to the € -tolerated 
sum of the hull and inaccuracies through positioning and object recognition, 
Equation 2.2 can be written as 


(l+¢)-(H-s+up,,): {1 tes )- 


Pixel 
(e-z H -S + Upos ee) H- S +U pos 
0 
U Pixel / A pixel A pixel U Pixel / Hii 

Equation 2.3 can then be solved for n, thus returning the minimum number of 
ZAC steps needed for automated switching from low magnification to high mag- 
nification. 

As a further restriction, the length reflected by one pixel in the lowest mag- 


nification (Ky / Ap;,.; ), has to be at least the same as the structural size s . Assuming 
that the < -factor is 10%, the positioning accuracy should be half the size s. 


(2.3) 
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Furthermore, if the object recognition accuracy is about 1% of the number of pixels 
(Api: = 500), then the number of ZAC steps is the ceiling of the value calculated 
with Equation 2.3, thus n = 2. The first magnification step (n =1) is then used for 
magnifying approx. 50 times, the second step (n=2) for magnifying again 
approx. two times. From this typical example, it can be concluded that for 
applications where in the first magnification the object is just recognized, two ZAC 
steps are generally necessary to reach the desired magnification. With very good 
parameter sets, zooming and centering can be achieved in one step. For poor object 
recognition accuracies and number of pixels, the number of zoom steps increases 
to approximately three. Hence, in typical nanohandling applications, the number 
of ZAC steps is smaller than or equal to three. 

Consequently, it should be considered that nanoscale handling processes con- 
tain more tasks compared to macroscale processes, as zoom-and-center steps 
occur more frequently. 


2.2.3 SEM-related Issues 


In the last section, the necessary number of ZAC steps was discussed. In the work 
space of SEMs, an increase in magnification implies a higher resolution, thus 
opening the possibility of substituting on-board position sensors by higher reso- 
lution SEM image acquisition and object recognition. This method has been used 
widely for the automation of nanohandling tasks [8, 9]. However, image acqui- 
sition takes longer than reading a sensor value; shorter image acquisition and 
processing times result in noisier images. Thus, the magnification where the on- 
board position sensor is replaced by object recognition has to be chosen taking the 
resolution enhancement as well as the delays into account. 


2.2.3.1 Sensor Resolution and Object Recognition 

The SEM is a high-resolution image acquisition unit and thus, as mentioned above, 
can be used as a sensor for closed-loop position control, substituting or supple- 
menting an on-board position sensor (Figure 2.2). SEM object recognition 
challenges are discussed in Chapter 4. Issues regarding the position controller are 
considered in Chapter 3. 

In Figure 2.3 a comparison between the resolution of a common on-board 
sensor for a linear axis and the achievable resolution using image processing and 
object recognition are plotted against the magnification of the SEM. It is clearly 
visible that already at magnification higher than 300 times in this case, the 
resolution achieved through object recognition is better than the on-board sensor 
resolution. Hence, with respect to image resolution, switching between the on- 
board position sensor and object recognition can occur when 


k 
Pa `U pixel 


Pixel 


<u (2.4) 
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where Ap ea is the number of pixels for the imaged square with edge length k, 
Upxe iS the accuracy of object recognition in pixels, and u is the sensor 
accuracy. 

Thus object recognition is preferably used compared to the on-board sensors, if 
the resolution of the object recognition system is significantly better. Modern 
piezoactuators, stick-slip, or continuous actuators accomplish step widths of 10 to 
20 nm. In most cases, therefore, the sensor resolution is the bottleneck rather 
than the actuator resolution. 


>) >| Controller | Actuator 


On-board sensor 


~ - 


`e— SEM Image Rec. o 


Figure 2.2. Typical control schematic for an actuator used for nanohandling automation. 
The on-board position sensor is substituted by SEM object recognition where the resolution 
is significantly better. 


Sensor 




















1000 


1004 


104 








sensor resolution [nm] 


—— SEM image recognition 
—<— on-board position sensor 








10 100 1000 10000 
SEM magnification @ 1024 x 768 pixels 


Figure 2.3. Comparison between the achievable sensor resolution using the on-board sensor 
of linear stick-slip axes and SEM object recognition. The on-board sensor is of type 
Numerik Jena L4 [10] and has an interpolated resolution of 50 nm, independent of the 
magnification. The resolution of SEM object recognition was measured according to the left 
side of Equation 2.4, using a recognition accuracy of 1 pixel. 
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2.2.3.2 Noise 

The SEM as a scanning image sensor has a resolution of approx. 0.1 nm for field 
emission guns. These high resolutions are only achievable with comparatively long 
image acquisition times, when the signal-to-noise ratio is maximized. In general, 
the image quality is improved when scanning speed is reduced. However, for most 
automation processes, high scanning speeds combined with fast object 
recognition are desirable. Therefore, a compromise between these two aspects has 
to be chosen. The typical noise of an SEM image plotted against the scanning time 
is shown in Figure 2.4. 

Obviously, it is necessary to increase the scan time if object recognition fails 
due to noisy images. For automation purposes, an update rate for sensor poses of 2 
per second is tolerable, but a lower rate significantly slows down the process. A 
reduction of noise by the factor 0.5 leads to approx. 10 times longer scan times, 
whereas a reduction of the scan time to one third leads to three times more noise in 
the image. Thus, specialized methods for high update rates at high recognition 
reliability are necessary, which will be described in Chapter 4. 


2.2.3.3 Velocity and Image Acquisition Time 

The scanning speed also limits the maximum travel speed of an actuator, when 
object recognition is used as sensor feedback. Consider a point-like object under 
observation of a vision sensor with a frame refresh rate f, , respectively a frame 
acquisition time T, and a resolution Apa» given in pixels. The area under 
observation is assumed to have an edge length of k. Then the object recognition 
accuracy u is determined by the following equation: 
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Figure 2.4. Relative noise of an SEM image for different scanning times at an image size of 
512 x 442 pixels. The relative noise was calculated as mean-square error relative to the error 
at a maximum scan time of 73 s. 
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-k 2 (2.5) 


U recognition A ` H Pixel > 
Pixel 


where Upe 18 a constant depending on the system and taking into account how 
precisely the position should be recognized. Typical values range from 1 to 10, 
e.g., a value of 2, using the sampling theorem. The scan process is simplified here 
to be a process where the image is scanned line by line and the column-wise 
scanning within a line is neglected. Then, two cases have to be considered for 
estimating the maximum allowed velocity Vmax of an object, if it has to be 
recognized at least in two subsequent images until the scanned area is left: 


e The object is moving orthogonal to the scanning direction, i.e., the time 
At between two occurrences of the object can be considered approx. T, . 

e The object is moving in the same direction as the scan is running. Then 
again two cases have to be considered: In the first, object movement and 
scan direction are anti-parallel. Then the time Aż until an object occurs in 
the following frame is shorter than the frame time T, . For the second, the 
object’s movement is parallel to the scan direction, resulting in longer time 
At between the occurrences in two successive frames. Then the maximum 
allowed velocity Vmax can be calculated from the intersection of two lines, 
resulting in 


1 7 U Pixel 


Vna = lrk, (2.6) 
IT: 


where T, is the image acquisition and object recognition time. Taking for 
example a scan time of 0.5 s (i.e., a sensor refresh rate of 2 Hz) at an edge 
length of the scan field of 50 um and a recognition accuracy of 10%, then 
the speed limit according to the above equation would be v,,,, = 45 m/s. 


2.3 Automated Nanohandling: Problems and Challenges 
2.3.1 Parasitic Forces 


The expression “parasitic forces” is commonly used as the collective term for 
surface forces that have major relevance in the micro- and nanoscale, i.e., van der 
Waals, electrostatic, and capillary forces. 

Electrostatic forces refer to forces due to electric charging of objects. Two 
objects that are charged with the same polarity repel each other, whereas oppo- 
sitional polarity leads to attraction. Typical causes for electrostatic forces in hand- 
ling processes are contact electrification, triboelectrification, and direct charging 
through the electron beam in the SEM. Charging through the electron beam of an 
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SEM can lead to repelling forces, resulting in objects floating around on the 
substrate surface. Other observed effects are electrostatic actuators in the SEM 
driven by the electron beam or image artifacts occurring due to the electrostatic 
deflection of the primary beam. In [11], the equation for estimating the electrostatic 
force between a sphere of radius r with a charge q and a conductive plane in 
distance is given by 


2 


q 

Fa 4ne(2ry” oi 
where ¢ is the dielectric permittivity. In practice, the force is hard to estimate 
because in general neither the charge nor the dielectric permittivity are known. In 
the SEM, special measures preventing or minimizing charging through the electron 
beam can be taken, e.g., observation in low-vacuum modus or optimization of the 
beam parameters. 

Van der Waals forces denote the (attractive) forces between atoms and mole- 
cules due to interatomic forces and can be calculated for a sphere on a plane by the 
following equation [11, 12]: 
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(2.8) 
where h is the Lifshitz-van der Waals constant and z is the atomic distance bet- 
ween the sphere and the plane. In [12], values for the Lifshitz-van der Waals con- 
stant are given for several material combinations, although this term is in general 
hard to estimate for handling processes. 

Capillary forces are due to liquid films between two objects. Even in high- 
vacuum chambers, e.g., when an SEM is used as vision sensor, liquid films on the 
surface of objects cannot be avoided. Due to water films (condensation), oil films 
(pump oil), etc., the surfaces of objects, tools, and substrate can never be 
considered dry. Estimations for the capillary force between a sphere and a plane 
are given in [11, 13], resulting in the following equation: 


F „,=4ary, (2.9) 


cap 


where y is the surface tension. However, the capillary force between object and 
gripper has also been used successfully for gripping, thus using the parasitic effect 
for handling objectives [14-16]. 

Fearing surveyed the parasitic forces and their influence on parts below 1 mm 
size. In [11], he suggested the following actions, among others, for reducing the 
influence of adhesion forces: 


1. Usage of conductive materials for reducing charging effects. In micro- and 
nanotechnology, silicon is, however, a very common material for handling 
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tools, which forms an insulating oxide. A work-around for this problem is 
covering the tools with a conductive layer, e.g., gold, where possible. 

2. Rough gripper jaw surfaces in order to minimize the contact area. This 
measure should even be adopted for the design of gripper jaws, which 
allow a minimum of point-point contacts between object and gripper. 
Furthermore, the gripper geometry should always be adapted to the object 
to be handled. 


It can be concluded from the survey of parasitic forces that most of them are very 
hard to estimate. Many factors in a handling system are either unknown or hardly 
measurable, e.g., the capillary forces due to condensed water in an SEM’s vacuum 
chamber. The complexity regarding geometry and interactions between multiple 
objects in a handling system additionally complicate the calculation of parasitic 
forces. Furthermore, these forces are time-variant, i.e., they can change drama- 
tically during a handling process. For example, a silicon gripper charged through 
the SEM’s electron beam can be discharged through contact with the substrate 
surface. 

Thus the parasitic forces are the major problem for the automation of handling 
tasks on the micro- and nanoscale due to their uncertainty and time-variance [16]. 
Experimental observations [17, 18] proved this conclusion. 


2.3.2 Contact Detection 


One of the major issues in the handling and assembly of nanoscale parts is the 
detection of contact between two objects. The issue arises when 2D images, e.g., 
from a light microscope or SEM, are used as global vision sensors determining the 
out-of-plane positions of objects relative to each other. Object recognition can be 
used for contact detection within the observed plane, but out-of-plane contact 
detection is not possible. A typical example is the detection of whether a gripper 
touches a probe surface. If this scene is observed from above, e.g., with an SEM 
providing high depth of focus, it is hardly possible to distinguish with common 
image recognition tools if the gripper touches the surface or not. Possible appro- 
aches for solving this problem are presented briefly in the following paragraphs. 

Depth from focus: The depth-from-focus method is described in [19, 20] for 
measuring the height difference between two objects by means of a focus sweep. 
For the series of images, two regions of interest containing both objects are 
defined. For every region and image, the variance is determined. The variance 
shows a local minimum in the variance function over the changing focus, where 
the object’s sharpness is at its maximum. Based on this method, the difference in 
height between two (object) surfaces can be determined. 

Touchdown sensor: This sensor [21-23] provides a method for measuring 
contact between two objects by means of a resonance method. The resonator 
consists of a piezoelectric actuator and a piezoelectric sensor, on which the tool, 
e.g., a gripper, is mounted. The piezoelectric actuator oscillates very close to the 
resonance frequency of the system. The system’s mechanical oscillation induces an 
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electrical current in the piezosensor. This current oscillates at the same frequency 
and with a certain amplitude. When the tool touches another object or a surface, the 
resonance frequency of the system changes, leading to a drop in the measured 
amplitude. 

3D vision: The idea of creating a 3D image out of two SEM images recorded 
from different angles has been the subject of research for several years. Especially 
with regard to automation processes, it is obligatory to deflect the electron beam 
instead of the probe. A promising approach will be presented in Chapter 5. 

Vision-based force measurement: Vision-based force measurement quantifies 
the deformation of a stressed object by means of object recognition [24, 25]. The 
algorithm can be applied for measuring forces — and thus contact — between two 
objects. For calculating the forces applied on an object, a priori knowledge is 
necessary, whereas for simple contact detection the deformation is evidence 
enough. However, this method is best applied where the object’s stiffness is low, 
e.g., measuring of contact or force between a gripper and a nanotube [26]. 


2.4 General Description of Assembly Processes 


In this section, the process design and considerations with regard to serial assembly 
tasks on the nanometer scale will be given. Figure 2.5 provides an overview of the 
typical tasks that are necessary to accomplish an assembly process. 

A description of the single tasks will be given in the next section. Based on this, 
further consideration regarding the reliability of assembly processes is given in the 
section after. Basically, simple handling tasks are the separation of an object, its 
transportation to another position and its release. Assembly processes comprehend 
(multiple) handling tasks, but are extended by joining processes and eventual 
inspection processes for quality assurance. 
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Figure 2.5. Overview of the tasks required in an assembly process. A typical process 
consists of the tasks “Separate”, “Transport”, and “Release”. Every task can again be 
separated into subtasks, which can be seen as primitives. 
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2.4.1 Description of the Single Tasks 


The tasks and subtasks forming a process can be best described as a change 
between two quantifiable states A and B. Especially for nanohandling tasks, it is 
very important that these states are quantifiable by means of measurement values, 
e.g., giving a position in coordinates or describing the contact between two objects. 
Hence, ideally, the process is always in a definite state. In the following sub- 
sections, elementary tasks will be described with respect to the special require- 
ments of the nano- and the macroworld. 

Separation: The separation task can be considered as the occasion when the 
object’s attachment status is changed, i.e., in the beginning its condition is “con- 
nected to substrate” and at the end of the process it is connected to a tool or an 
intermediate product. The task itself can be achieved by several methods, e.g., 
gripping and lifting, gluing and etching. Already from these examples it is obvious 
that the separation task includes at least two subtasks, i.e., connecting the object to 
the tool and releasing the object from the substrate. 

The separation of objects is one of the most difficult tasks in the field of 
nanohandling (Figure 2.6). The main reasons are the influences of parasitic forces 
and the comparatively weak forces which can be exerted by grippers or similar 
tools. An overview of strategies for lifting off small objects on the nanoscale, e.g., 
nanowires, is given in [6, 17]. 

Transport: The transport task differs from macroscale transport tasks simply 
by the possible number of orders of magnitude in geometric scale, but is concep- 
tually the same: the gripped or previously fixed object is transported from position 
A to a position B. Care has to be taken that the object is not released by accident 
during transport, e.g., caused through vibrations of the actuators. A further 
description of positioning issues and position control will be given in Chapter 3 
and Section 2.6. 

Release: The “release” task is the reverse of “separation”, i.e., at first the object 
is attached to the substrate and then the object is detached from the tool. The 
parasitic forces cause an object on the nanoscale to stick to handling tools until the 
forces between the surface, where the object should be placed, are higher than the 
forces between the tool and the object. A reduction in sticking forces can be 
achieved using two principal ways: 


1. The sticking force is reduced by reducing the contact size, e.g., through 
specially formed gripper jaws or by reducing the influence of the parasitic 
forces, e.g., through special coatings on the jaws. These measures usually 
require a considerable technical effort. 

2. Instead of adapting the gripper to the gripping task, a special technique for 
releasing the object can be applied, e.g., wiping off [17] or shaking off. 
Both techniques aim at shifting the balance between gripper-object forces 
and object-substrate forces to the substrate side. 
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Joining: The joining of objects is the central task for assembly processes. In 
principle, three different methods can be used for joining objects: 


1. Material closure: two objects are connected using a material connection 
between both, e.g., welding, gluing, or soldering. In Chapter 10, electron 
beam induced deposition (EBiD) will be explained in more detail, as it is a 
very promising method for joining parts through material closure inside the 
SEM. 

2. Force closure: two objects are connected by a force, which can also be a 
parasitic force [27]. 

3. Form closure: two objects are joined by their geometry. This approach is 
not very common in nanoassembly tasks. 


500um ó 





Figure 2.6. Typical objects to be manipulated in micro- and nanohandling tasks. a. A 
gripper trying to grab one out of a bunch of silicon nanowires in the SEM. The nanowires 
with a diameter between 200 and 500 nm and a length of several micrometers have been put 
on the substrate simply by peeling off. This separation task is very hard to automate, due to 
the parasitic forces holding the nanowires together. b. A glass ball (diameter approx. 30 um) 
has to be gripped by a silicon gripper in the light microscope. This task can be automated, 
because the balls are split up and do not adhere to the surface due to the reduced contact 
area. c. CNTs grown in a matrix on a silicon wafer. This is a good starting point for 
automation processes, as the nanowires are separated and orientated on the wafer. However, 
they have to be detached from the substrate by breaking or etching. 
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Inspection: The inspection task serves for quality assurance. Many parameters can 
be tested to prove that the assembly process has been successful: stressing the 
connection up to a threshold force, chemical analysis of deposited material, elec- 
trical characterization of the bonding through resistance measurements. 


2.4.2 General Flowchart of Handling Processes 


Based on the simple tasks described above, it is possible to set up more complex 
processes, e.g., handling and assembly by combining these to a linked task chain. 
Generally, every process, task, and subtask can be described in the process flow 
chart as a change from condition A to condition B. The conditions can be described 
as a vector containing the position data of single components, i.e., objects and 
tools, and their relation to each other, e.g., “part 1 connected to part 2”. Based on 
these measurable values, it is possible to trace failures that would result in a 
process failure. Figure 2.7 shows the main tasks needed for bonding a CNT to an 
AFM tip. Additionally, the number of subtasks needed for successfully fulfilling a 
task is given. 


2.5 Approaches for Improving Reliability and Throughput 
2.5.1 Improving Reliability 


The reliability of the overall process R for a series of subtasks can be calcu- 


process 


lated by multiplying the single subtask reliabilities, i.e., 
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Figure 2.7. Typical process layout for bonding a single CNT to an AFM tip, showing the 
conditions between the tasks, the tasks and the number of subtasks for every task 
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where Fasas ; ÍS the reliability of the i-th subtask and n is the amount of subtasks. 
For example, a mean reliability r.,,,,, Of 98% for the 23 subtasks shown in Figure 
2.7, leads to an over-all reliability for the process of 63%. A further decrease of the 
mean reliability for the subtasks down to 95% reduces the over-all reliability to 
31%. This example shows the importance of maximizing the subtask reliability on 
the one hand and minimizing the number of subtasks on the other. 

Another means, which is of special interest for serial assembly processes, is the 
definition of fallback markers in the process chain. If a task or a subtask fails, the 
system should be brought into a defined state, from where the chain can be con- 
tinued. This method is referred to as failure analysis with non-ambiguous 
retrace. 





Minimizing the number of subtasks: For minimizing the number of subtasks, 
several measures can be adopted: 


1. Skillfully planning the handling tasks. This can be achieved through 
reducing the number of tools needed. One very important consideration, 
which should be taken into account for assembly tasks, is, e.g., the 
omission of a gripper. This reduces the number of tasks substantially, 
because intermediate tasks, which are not directly concerned with 
connecting two objects to each other, can be left out. For example, 
connecting CNTs to AFM tips is a process where the gripper can be left 
out, if the CNTs come in a suitable pre-packaged orientation (Figure 2.6c). 

2. Optimizing the number of subtasks needed for fulfilling a task. A typical 
example is minimizing the number of necessary ZAC steps (Section 2.2.2). 
Further improvements can be achieved if, e.g., position sensors providing 
very high resolution over a wide scale are used. This prevents the switching 
of sensors and thus leads to a reduction of subtasks for the same 
positioning task. 


Maximizing subtask reliability: subtask reliability can be increased using the 
following measures: 


1. Continuous application of sensors, ideally setting up of closed-loop control 
systems, in order to trap exceptions. Jn situ measurement methods are of 
special interest for controlling subtasks. 

2. Attaching and detaching subtasks are of special interest in terms of reli- 
ability. Based on the indetermination of parasitic forces, form and force 
closure should be substituted through material closure where possible, e.g., 
bonding TEM lamellae to tips instead of gripping them with mechanical 


grippers. 
2.5.2 Improving Throughput 


The throughput D of the process can be defined as the inverse of the mean time 
needed for one process T 


process ? 


i.e., 
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process 


For estimating the influence of single parameters, the following assumptions are 
made: a process consists of n subtasks, with a mean duration Tisas and a mean 
reliability 7.4/4, each. Every subtask is repeated until it has succeeded and thus the 
whole process is successful. For a large number of equal processes, the mean 
process duration can then be calculated from 











| n: T nbiask ; (2. 12) 
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From this equation, the influence of the single parameters on the throughput can be 
qualified. For maximizing the throughput of a process, it is thus necessary to 
minimize the number of subtasks and to maximize the subtask reliability. Both 
measures have already been discussed in the section above. However, for mini- 
mizing the mean duration Tsa Of a subtask, the following measures can be 
taken: 





1. Optimizing the travel speed of actuated parts through fast sensors. 
Additionally, as scanning vision sensors are widely used for nanohandling, 
the optimization of image acquisition and recognition has to be taken into 
account. 

2. Especially for handling tasks, where the separation task is often very 
critical, the duration can be minimized by optimizing the layout of the 
stored objects. 

3. Contact detection can be one of the very time-consuming tasks, because the 
travel speed of the actuators has to be reduced, in order to prevent hard 
crashes. Therefore, vision-based methods for the determination of the 
distance between object and tool, respectively substrate, are preferred. One 
approach, 3D vision sensors, is discussed in Chapter 5. 

4. Optimizing controllers for speed, as discussed in Chapter 3. 


2.6 Automated Microrobot-based Nanohandling Station 


The automated microrobot-based nanohandling station for TEM lamellae hand- 
ling was one of the first implementations of the generic AMNS concept presented 
in Chapter 1. The station was developed by the Division Microrobotics and Control 
Engineering (AMiR), University of Oldenburg, in the framework of the EU project 
ROBOSEM (grant number GRD1-2001-41861). The main purpose of the project 
was the integration of microrobots as well as position and force sensors into the 
vacuum chamber of an SEM. The client-server-based control system supports the 
user during nanohandling processes, where the objects’ sizes range from some 
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hundred um to some hundred nm. A good example that has evoked interest from 
industrial partners is the handling of silicon lamellae that are to be evaluated in a 
TEM. 


2.6.1 AMNS Components 


2.6.1.1 Setup 

The setup of the nanohandling station (schematic in Figure 2.8 and picture in 
Figure 2.9) consists of one microrobot that positions the sample to be handled 
(“sample robot”), and one microrobot that positions the end-effector, performing 
the actual handling task (“handling robot”) [9, 28, 29]. Besides the main sensor, the 
SEM with image processing, CCD cameras with image processing, a position 
sensor, and a sensor for contact detection (“touchDown sensor”) are employed to 
support the user and to allow for automatic positioning. The setup is attached to an 
exchangeable door of the SEM, so that it can be assembled, maintained and tested 
outside the vacuum chamber with a light microscope as SEM replacement. This 
reduces valuable SEM time, avoids pump time for the vacuum and generally 
allows for easy access to all components. 


SEM end-effector 
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touchdown sensor 
globe camera 


manipulator 





globe platform 


t latfi 
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optical sensor 


linear axis 





X 
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Figure 2.8. Schematic of the nanohandling station. The sample robot consists of the stage 
platform with a sphere carrying the specimen. It can position the sample in all six DoF. The 
handling robot consists of the effector platform with the manipulator carrying the 
touchdown sensor and a gripper. It can position the end-effector in the three DoF of a 
horizontal plane. The two translational DoF have a high actuation resolution. 
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Figure 2.9. Setup of the nanohandling station. For development purposes, a light micro- 
scope is used and removed when the station is used in the SEM. 


2.6.1.2 Actuators 
The sample robot consists of two single-disk mobile platforms (diameter: 30 mm) 
and a linear axis. One mobile platform (stage platform in Figure 2.8) moves on a 
horizontal glass plate and carries another platform (globe platform), which is 
mounted upside down to rotate the sphere-shaped sample holder in all three rota- 
tional degrees of freedom (DoF). The working principle of the mobile platforms is 
explained below. The mobile platforms holding the sample can be moved vertically 
with a piezo-based linear axis from [30], which is fixed to the SEM door. Using 
all components of the sample robot, the sample can be positioned in all six DoF. 
The handling robot consists of a triple-disk mobile platform (diameter: 
60 mm), which carries a manipulator with an end-effector, e.g., a gripper. The 
mobile platform (effector platform in Figure 2.8) moves on a separate horizontal 
glass plate around the sample robot to coarse-position the end-effector, and the 
manipulator then positions the end-effector with higher resolution to its desired 
(x, y) -position. The manipulator consists of two piezo stack actuators, which drive 
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leverage arms fixed by flexible hinges. The maximum stroke of the table is about 
40 um. The end-effector can thus be positioned in x and y with high resolution, 
and rotated around the z-axis. The end-effector itself can be passive, like an STM 
(scanning tunneling microscope) tip, or an AFM cantilever, or active like a 
microgripper from [31], or from the Technical University of Denmark, Lyngby, 
Denmark (Chapter 7). 


2.6.1.3 Mobile Microrobots 

Two different implementations of a mobile microrobot platform are integrated into 
the AMNS [32-34]. The triple-disk platform is actuated by three piezodisks, 
which are each segmented into three parts (Figure 2.10a and c). A small ruby bead 
is glued to each segment, and each three-tuple of ruby beads drives one of three 
metal or sapphire spheres, which support the mobile platform. Instead of three 
piezodisks with three segments each, the single-disk platform consists of one 
piezodisk with nine segments (Figure 2.10b and d). In the setup, the triple-disk 
platform implements the effector platform while the single-disk platform imple- 
ments the stage platform and the globe platform. 
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Figure 2.10. The triple-disk platform with three piezodisks: a. bottom view and c. channel 
configuration. The single-disk platform with one piezodisk: b. bottom view and d. channel 
configuration. 
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The developed platforms make use of the stick-slip principle by applying a vol- 
tage signal, which consists of a part with a gentle slope and a part with a steep 
slope, such that the segments are bent correspondingly slowly and fast (Figure 
2.11). During the slow bending, the small ruby bead moves the large sapphire 
sphere, leading to a small rotation. This is called the stick phase. During the fast 
bending, the small ruby bead slides over the sapphire sphere, which therefore keeps 
its orientation. This is called the slip phase. 

The number of control channels is reduced from nine to six by electrically 
connecting each of three piezo segment pairs. The configurations given in Figure 
2.10c and Figure 2.10d yield three principal translation directions and one principle 
rotation direction, such that the microrobot platforms can move in all three degrees 
of freedom. 


2.6.1.4 Sensors 

The main high-resolution sensor of the nanohandling station is a LEO 1450 SEM 
[35], in combination with image processing, which provides “poses” of micro- and 
nanoobjects and end-effectors with resolutions down to 2nm (magnification 
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Figure 2.11. a. Typical voltage signal that is applied to a piezo segment. b. Stick phase by 
slow deformation and slip phase by fast deformation of the piezo segment. 
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50,000x, fast scanning). The generated pictures are acquired using the digital 
image acquisition unit offered by [36] and are processed using algorithms that have 
been developed at the AMiR [37-39]. For the coarse positioning of the mobile 
platforms, three corresponding CCD cameras are mounted on the SEM door. 
Together with the image processing software developed at the AMiR, they measure 
the mobile platform’s poses with resolutions of about 60 um (stage camera) and 
170 um (effector camera). 

Measurements in the vertical direction are performed with two sensors. An 
optical position sensor from [40] measures the z-position of the linear axis with 
respect to a fixed reference (resolution below 1 um). In addition, a touchdown 
sensor detects when the end-effector touches another object, e.g., a microobject to 
be handled [9]. The touchdown sensor is a bimorph piezo-bending actuator, which 
is attached to the manipulator, and acts as a cantilever holding the end-effector. 
One ceramic layer is driven by an AC voltage with small amplitude (5 mV), and 
the other layer measures the amplitude of the resulting mechanical oscillation 
(approx. 50 nm). A contact between end-effector and microobject then results in a 
considerable and distinct drop in the measured amplitude. 


2.6.1.5 Control Architecture 

The control system is set up as client-server architecture with communication over 
TCP/IP to allow for flexible use of the control and sensor modules in different 
applications (Figure 2.12). 
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Figure 2.12. Control system architecture of the nanohandling station (a detailed view of the 
inlay figure in given in Figure 2.8). The touchdown server, the vision server, and the 
position server send their measurement data to the sensor server. The control server requests 
the measurement data when needed to control the actuators. The super-client master control 
controls all servers remotely and provides an interface to the user. 
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On the sensor side, there is a vision server that is responsible for the acquisition 
and processing of images from the SEM and from the CCD cameras. Its task is to 
detect features of microobjects, end-effectors, or microrobots and to determine 
their pose in a global frame (Chapter 4). Also, the optical position sensor and the 
touchdown sensor have their own server applications (touchdown server and 
position server). All servers continuously send their data to a sensor server, 
which stores the most recent data of each sensor and which provides that data to 
any client requesting it. On the actuation side, a low-level control server is 
responsible for the access of the actuation hardware and for the execution of 
control primitives with sensor feedback via the sensor server. The most common 
process primitive is positioning an object like an end-effector with feedback from a 
microscope. The master control module serves as a super-client and controls all 
servers remotely. For the vision servers, it remotely chooses between different 
tracking models and input sources, and starts and stops the tracking and defines 
regions of interest. The touchdown server and the position server can be switched 
on and off remotely. For monitoring, the master control module continuously re- 
quests data from the sensor server about the microrobots’ poses. On the control 
server, it remotely starts and stops the execution of low-level control primitives 
and forwards data from a teleoperation device. The user can control the micro- 
robots via teleoperation or in a semi-automated way by triggering process pri- 
mitives. The user receives status and position feedback from the graphical user 
interface of the high-level control module or visual feedback from the vision 
servers. 


2.6.1.6 User Interface 

The user can interact with the AMNS via a graphical user interface (GUI) and via 
a teleoperation device. In addition, an emergency stop button stops all actuators 
by disconnecting them from the power supply. 

The GUI is the main part of the master control module and allows control of 
the connection to the servers and automation, supports teleoperation and gives 
information for monitoring (Figure 2.13). The connection part effects connection 
to, and disconnection from, the servers. The servers are identified by their IP add- 
resses and their constant port numbers. The automation part effects the triggering 
of a predefined sequence of process primitives. Depending on the current 
automation state, it is possible to start this sequence at different intermediate states. 
An additional calibration provides a comfortable read-out and saving of different 
desired poses. For example, the pose of the effector platform, at which the 
connected gripper is in the SEM's field of view, can be reached by teleoperating 
the platform accordingly. The current pose can then be used by the automation 
module as the desired pose when coarse-positioning the gripper. The pose part can 
show the pose values of all sensors, which are connected to the sensor server. The 
teleoperation and sensor control part shows the current sensor and the current 
actuator and allows changing them. Controlled by a cordless gamepad, the current 
actuator can be moved in any desired direction and with any desired velocity. The 
maximum velocity can be adjusted in a wide range from constant movement to a 
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single step. Furthermore, the current sensor and actuator can be changed by the 
gamepad. Finally, in the system messages part, there is a textual output about the 
current and the last tasks. 


2.6.2 Experimental Setup: Handling of TEM Lamellae 


The control system can be seen as a tool to support user-performed nanohandling 
tasks in a teleoperated or in an automatic mode. Within the framework of the 
ROBOSEM project, the nanohandling station has been used to demonstrate the 
semi-automated handling of lamellae. 

The handling sequence comprises the automated and teleoperated positioning 
of mobile microrobots. The names of actuators and sensors of the following 
sequence steps refer to the description above: 
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Figure 2.13. Graphical user interface of the AMNS 
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e Lamella selection. The sample, a silicon chip with four lamellae, is placed 
on the sphere, which can be rotated by the globe platform. First, the sample 
is coarse-positioned into the SEM’s field of vision by moving the stage 
platform to a predefined pose with respect to the stage camera. The user 
then selects one of the lamellae, which is detected in the SEM image 
(Figure 2.14a). 

e Automatic lamella positioning. The selected lamella is successively fine- 
positioned at a pose that allows good gripping with SEM feedback at 
magnifications of 50x, 300x, and 1000x (Figure 2.14b, c, and d). 

e Automatic gripper positioning. To avoid a collision during the gripper 
positioning, the lamella is lowered by some 100 um using the linear axis 
with feedback from the optical sensor. The gripper is then coarse- 
positioned into the SEM’s field of vision by moving the effector platform 
to a predefined pose with respect to the effector camera, followed by a 
successive fine-positioning with SEM feedback at magnifications of 50x 
and 300x (Figure 2.15a and b). The most accurate positioning of the 
gripper is then achieved by using the manipulator with feedback from the 
SEM. For gripping, the lamella is lifted up again, until the touchdown 
sensor detects a contact between gripper and specimen (Figure 2.15c). 

e Lamella gripping. The lamella is gripped in teleoperation mode since the 
mismatch of the gripper’s stiffness and the lamella’s pull-off force leads to 
low gripping reliability (Figure 2.15d). 

The lamella can then be transported to a TEM grid, which is placed on the same 


sample holder. The sequence is similar to the gripping sequence, i.e., the 


positioning of TEM grid and gripper can be done automatically with feedback from 


Table 2.2. Positioning accuracies during different tasks when handling lamellae 








Task Actuator Meas. frame Positioning accuracy 
x y Z QP 
[um] [um] [um] [°] 
lamella Bees Se stage platf. Stage camera 20 20 0.5 
coarse positioning 
lamella 
fine positioning stage platf. SEM (world) 2 2 
ae effector platf. Erect 20 20 0.5 
coarse positioning camera 
Bee ties effector platf. SEM (world) 1 1 
fine positioning 
Se a. manipulator SEM (world) 0.5 0.5 
fine positioning 
lamela linear axis optical sensor 1 


coarse positioning 
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Figure 2.14. a. Selection of one of four lamellae. Successive positioning of the selected 
lamella with SEM feedback with magnifications of b. 50x, c. 300x and d. 1000x. Size of the 
lamella: 20 um x 10 um x 100 nm. 


cameras, SEM, and touchdown sensor, while the actual release process must be 
carried out by teleoperation. 

Table 2.2 lists the achieved positioning accuracies during different tasks when 
handling TEM lamellae. The accuracies are given with respect to the sensors which 
provide the feedback. For example, the gripper that is attached to the effector 
platform is positioned with an accuracy of 1 um with respect to the SEM (gripper 
fine positioning in Table 2.2). The world frame is set to be the SEM image frame 
such that poses for the gripper and for the lamella can be used without any trans- 
formations. Moreover, the positioning accuracies that are given in Table 2.2 are the 
thresholds for the controller to stop the corresponding positioning process. They 
are chosen rather conservatively to increase the overall robustness and speed of the 
handling process. 
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Figure 2.15. Positioning of the gripper with SEM feedback with magnifications of a. 50x 
and b. 300x. c. lift-up of the lamella with feedback from touchdown sensor. d. teleoperated 
gripping of the lamella. Gripper opening: 50 um. 


2.7 Conclusions 


Vision sensors are essential for nanohandling tasks. A reasonable choice due to its 
good resolution, reasonable image acquisition times, and scalability is the 
scanning electron microscope. The SEM provides the possibility of substituting 
on-board position sensors and thus extending the closed-loop positioning 
resolution of a sensor-actuator system. The seamless zooming over several orders 
of magnitude is especially advantageous for automation, because typically more 
than one zoom-and-center step is necessary for accomplishing a handling or 
assembly task on the nanoscale. The number of necessary ZAC steps can be 
quantified to determine and optimize the overall number of process tasks. When 
applying SEM-based pose measurement in a closed-loop control structure, 
however, there is a trade-off between long image acquisition times and noisy 
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images. The image acquisition time determines the maximum allowable velocity of 
the actuators and the noise determines the reliability of the object tracking. 

There are two major challenges regarding automated nanohandling. Firstly, 
inevitable and time-variant parasitic forces result in ambiguous behavior of 
objects. Possible approaches are the avoidance of grippers and the performance of 
handling tasks with material closure between object, tool, and substrate. Secondly, 
a reliable method for the automatic detection of contact between tools, objects, 
and substrate must be found. Possible approaches are depth-from-focus methods, 
the touchdown sensor concept, 3D SEM vision, and vision-based force 
measurement. 

For the optimization of automated nanohandling and assembly processes, the 
reliability of each task can be improved. This can be accomplished by reducing 
the number of tasks and subtasks, by maximizing each subtask's reliability (closed- 
loop control, constant monitoring, in situ measurement), and by applying material 
closure for attaching and detaching subtasks. Furthermore, the throughput can be 
improved by prearranging parts for an optimized separation, by using a fast method 
for contact detection, and by maximizing the velocity of actuated parts. 

Finally, the concept of an automated microrobot-based nanohandling 
station has been demonstrated for the handling of TEM lamellae, which integrates 
all essential components named above. An SEM and CCD cameras, together with 
dedicated image processing, are used as position sensors, which provide the 
feedback for the closed-loop control of different microrobots. Mobile platforms are 
applied for coarse positioning and piezostack actuators for fine positioning. A 
touchdown sensor detects contact between gripper and substrate. The commu- 
nication framework for the sensors and actuators is a designed TCP/IP-based dis- 
tributed control system. The separation, transportation, and release tasks are 
implemented, although separation proved to be difficult due to the parasitic forces, 
to the low gripping force, and to the low stiffness of the gripper. 

The future activities aim at improving reliability and throughput for auto- 
mated handling on the nanoscale. EBiD will be used as a reliable joining techni- 
que, avoiding grippers where possible. A very important step towards reliable pro- 
cesses on the nanoscale is the application of subtask failure analysis and non- 
ambiguous retrace, as well as a final inspection task for the quality assurance. 
Different contact detection methods will be evaluated with respect to reliability and 
speed. The communication framework is currently being redesigned and will use 
the common object request broker architecture (CORBA), which allows the inte- 
gration of modules on different platforms and written in different programming 
languages. Finally, a script-language-based high-level controller allows flexible 
execution of different nanohandling processes with the same system (Chapter 7). 
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3.1 Introduction 


3.1.1 Control of Mobile Microrobots 


The mobile microrobots developed by AMiR and described in the previous chap- 
ter are controlled automatically or via teleoperation. Feedback during these proc- 
esses is provided by a global pose sensor, which could be a scanning electron mi- 
croscope (SEM), a light microscope or a video camera. As described in the next 
section, the mobile microrobot’s pose controller contains several sequential sub- 
tasks that are performed before the signals that are applied to the microrobot’s 
actuators are determined. These subtasks are either performed by a computer in 
automatic mode or by a human being in teleoperation mode (Figure 3.1), except for 
the actuator controller, which is typically implemented by an electronic device or a 
computer. 
system 


i : behaviour 
desired actuation 


velocity m parameters s 







d 
vV 





actuator 


teleoperation 
P controller 


or 
automatic 
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measured velocity v” 





Figure 3.1. Actuator controller during teleoperation or automatic control 
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The actuator controller maps from a desired velocity vector v“ to a parameter 
vector p“, which defines the signals that drive the microrobot. An exact descrip- 
tion of the actuator controller, however, is difficult to achieve because the mobile 
microrobot’s motion behavior, i.e., of the mapping s from actuation parameters 
p° to a measured velocity v”, shows some unwanted properties. Generally, it 
requires a relatively high effort to exactly determine all parameters of a mobile 
microrobot’s parameterized analytical model. The problems are variations of the 
parameters between different instances [1], complex measurements [2, 3], and 
variations of the parameters over time due to their dependency on environmental 
conditions like humidity and temperature [4, 5], and on the internal state of the 
microrobot like wear [6], piezo-drift [7], etc. Moreover, the microrobot’s inverse 
behavior s™', and thus an actuator controller model m , is often not unambiguously 
defined, i.e., different actuation parameters p° lead to the same motion v” [8, 9]. 
In the case of the considered mobile microrobots, there are six control channels for 
three degrees of freedom (DoF) of motion. 

Different approaches are followed to determine the actuator controller model of 
currently existing mobile microrobots. For the Abalone platform, the parameters of 
an analytical model are found by measurements [2]. For the MINIMAN platform 
[10], the pose is either controlled by a neural network controller or a fuzzy logic 
controller [1]. The parameters for the approximate analytical model of the Mi- 
CRON platform [11, 12] are estimated analytically, with help of neural networks 
[13] or with genetic programming [14]. The actuator controller model for the Ao- 
yama Labs platform [15] can be found relatively easily due to the inch-worm 
locomotion principle and the actuator configuration. For the NanoWalker plat- 
form, a parameterized model of the motion behavior is briefly described in [16], 
but it is not used for the derivation of an actuator controller. 

In all cases a transfer of the developed methods to other application domains, 
and in particular to other mobile microrobots, is quite difficult, because assump- 
tions are made that rely on the specialties of each mobile microrobot. Moreover, an 
incremental adaptation to a changing system behavior has not been performed by 
any of the approaches. 


3.1.2 Self-organizing Map as Inverse Model Controller 


In a control engineering context, an inverse model maps from a desired system 
output g“ to a system input p° such that the measured system output g” is iden- 
tical to the desired one (Figure 3.2a). In a closed-loop controller, an inverse model 
represents the final control element, sometimes referred to as linearization, with an 
inverse characteristic of the system (Figure 3.2b). A mobile microrobot is an ex- 
ample where an inverse model maps from desired velocities g” = v” to actuation 
parameters p°, such that the measured velocities g” =v” are as desired. A 
closed-loop controller then determines desired velocities v” from the difference 
between desired poses s” and measured poses s”. 
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Figure 3.2. Inverse model in a. an open-loop control system and b. a closed-loop control 
system 
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Self-organizing maps (SOM) [17, 18] are a type of artificial neural network 
that learns unsupervised, i.e., without an explicit teacher, and that are mainly used 
for data visualization and classification problems [19, 20]. There are some ap- 
proaches, though, where SOM-based models are used in a control setup with su- 
pervised learning, resulting in so-called "self-supervised" models [21]. 

SOM consists of numbered nodes n, ,i= 1...|N | . The nodes are arranged in a 
topology that is defined by associated support vectors ¢, in a topology space C. 
This topology can be one- or multidimensional and can have different shapes such 
as a rectangular grid (Figure 3.3) or a hexagonal grid. The neighborhood relation 
between two nodes n, and n, is expressed as a distance measure between their 
associated topology support vectors c, ande,. Each node n, also has an input 
support vector g, in an input space G and a corresponding output support vector 
p; in an output space P associated to it, which is why this arrangement is called 
associative memory or associative map. SOM has the following two functions in 
the context of modeling and control: 


1. Desired system outputs g are mapped to corresponding inputs 
p= m(g") of the system to be controlled (Figure 3.3a). 

2. System information, which is represented by pairs of training system inputs 
p‘ and measured system outputs g” = s(p'). is used to learn the map- 
ping, such that the error m(g” —p’ is minimized (Figure 3.3b). 
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input space G topology space C output space P 





input space G topology space C output space P 





Figure 3.3. a. Mapping with a SOM and self-organization in input space. b. Learning of 
inverse system behavior by self-organization in output space. 


The basic SOM approach shows some interesting properties for the implemen- 
tation of a learning inverse model controller, as well as contrasting with classic 
artificial neural networks such as multilayer perceptron networks [22]. Firstly, 
since each node is exclusively responsible for a local area of the input space, the 
behavior of the input-output map is interpretable by human beings [18]. More 
important for the application to inverse model controllers, though, is the self- 
organization aspect of the SOM learning rule, where SOM’s look-up-table-based 
data is ordered according to the pre-defined arrangement of topology support vec- 
tors [18]. Assuming that the map m is continuous, its inverse m`, which ap- 
proximates a continuous system behavior s, becomes continuous, too [23, 24]. 
Moreover, the ordering process can be seen as an optimization process, which 
resolves possible ambiguities when learning the map m [25]. SOM provided a 
good starting point for the development of enhanced mapping and learning ap- 
proaches for robotics and control. The most important are summarized in Table 3.1 
The local linear map (LLM) [25] was the first complete SOM-based algorithm 
framework for robot control [21, 26-31]. Its local linear models allow for extrapo- 
lation outside the convex hull of the support vectors and provide a means for the 
estimation of output support vectors, such that the mapping error is minimized. 
Nevertheless, the local linear model’s dependency on the inverse model’s Jacobi- 
ans makes an interpretation or an initialization with a-priori knowledge difficult. 
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Moreover, the sometimes global influence of the learning rule and the time- 
dependency of the learning parameters are not suitable for applications where the 
map is learned during operation and successive input vectors are correlated. Fur- 
thermore, LLM shows discontinuities at the borders of the nodes’ influence range. 
By using the continuous SOM (CSOM) [31, 32] interpolation algorithm as an 
extension of LLM, these discontinuities can be avoided. The parameterized SOM 
(PSOM) [33, 34] does not model the inverse system behavior but models the sys- 
tem behavior itself and finds an optimal inverse during runtime. It provides a con- 
tinuous and interpretable mapping, which is based on polynomials. Depending on 
the configuration of the map, these polynomials can lead to oscillations during 
interpolation and strong distortions during extrapolation [34]. PSOM inherits the 
learning properties of SOM, i.e., time-dependence of locality, etc. The ambiguity 
resolution of PSOM is flexible and can even change its optimization criterion "on- 
the-fly", but for the price that the whole system behavior must be modeled. The last 
interesting approach is the continuously interpolating SOM (CI-SOM) [35-38], 
which mainly implements a continuous interpolation for SOM. The resulting map- 
ping is continuous and local, only depends on the support vectors, and performs a 
meaningful extrapolation. The only drawback is the dependency on application- 
and configuration-specific interpolation coefficients. The focus of this develop- 
ment is the continuous mapping. The learning is probably performed by the SOM- 
learning rule, but detailed information on that topic was not available. 


Table 3.1. Comparison of features of different SOM-based methods 











Feature SOM LLM CSOM PSOM CI-SOM SOLIM 
Mapping 

Continuity = = x x x x 
Locality x x x x/— x x 
Interpretability x = — x x x 
Extrapolation = x x x/— x x 
Learning 

Locality x/— x/— x/— x/— [x/-] x 
Time-indep. params. - - - - [H] x 
Ambiguity resolution x x x x/— [x] x 





The analysis of existing approaches shows that there is no SOM-based ap- 
proach that can directly implement a learning inverse model controller, which can 
learn incrementally. The self-organizing locally interpolating map (SOLIM) ap- 
proach suggested in Section 3.3 tries to close this gap by providing a method for a 
continuous, local, and interpretable mapping that also performs a meaningful extra- 
polation, and to provide a new learning method that is local, does not use time- 
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dependent learning coefficients, and that exploits the neighborhood relation for the 
resolution of the inverse model’s ambiguities. Furthermore, the SOLIM algorithm 
framework is easy to use, i.e., there are as few parameters as possible to be ad- 
justed by the user. As shown in the following sections, this goal has been reached 
to a certain extent. However, the topology must still be provided by the user. 


3.2 Closed-loop Pose Control 


As shown in Chapter 2, the control structure of the automated microrobot-based 
nanohandling station (AMNS) consists of a control server, which is responsible for 
the execution of process primitives, and a master control program, which is respon- 
sible for the sequential calling and monitoring of the process primitives. The most 
basic process primitive of the AMNS is to change the measured pose s” of a plat- 
form, microobject, or end-effector from a given starting pose s* to a desired pose 
s” along a linear trajectory. For every process primitive, the control server gener- 
ates signals that drive the microrobots’ actuators accordingly. The parameters of 
the signals are varied, depending on the feedback from a sensor or depending on 
the input from a teleoperation interface, where the user interprets the data from the 
sensors. In Figure 3.4 the control structure used is shown with a trajectory control- 
ler, a motion controller and an actuator controller, which together constitute the 
implemented low-level controller. 


desired desired desired actuation 


global pose local pose local velocity 4 parameters 
d ? i : d 
Swe traject Svs} motion | Vie | actuator | p° 
controller < controller controller 
Lp 
—_— pose Swc ( measured velocity Vje 


Figure 3.4. Low-level controller consisting of a trajectory controller, a motion controller, 
































and an actuation controller 
3.2.1 Pose and Velocity 


The pose vectors s used in this chapter are either defined in Cartesian coordinates 
or in polar coordinates, indicated by the subscripts p and c, respectively. Further- 
more, they are either given with respect to a world coordinate system or with re- 
spect to a local, moving coordinate system, indicated by additional subscripts w 
and /, respectively. The same notation applies for velocity vectors v. A Cartesian 
pose vector s, is described as (x,y,g) with its components in the x-direction, in 
the y-direction and the angle g around the z-axis. Correspondingly, a Cartesian 
velocity vector v, is the first derivative (x, ¥.@) of a pose vector. A polar pose 
vector s, is described by (r,2,~) with the distance r= sqrt (x? + y’), the direc- 
tion z=atan2(y/x) and the same angle g.atan2(y/x) yields the unambiguous 
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angle of the vector (x,y) by modulating the sign of tan(j/x) , depending on the 
sign of y and x. A polar velocity vector v, is described accordingly by (v,2,@) 
with the velocity v= sqrt (x? x ty ae the direction 1=atan2(j/x) and the angular 
velocity @=@ . Conversely, the components of Cartesian pose and velocity vec- 
tors can be easily determined as x =rcos(z) and y=rsin(z), and as x = vcos(z) 
and y =vsin(z) , respectively. 


3.2.2 Trajectory Controller 


The trajectory controller (Figure 3.4 and Figure 3.5 — left side) determines a local 
deviation s;, from a linear trajectory and an orthogonal desired local pose s;,_, 
along that trajectory. These two poses correspond to the two distinct objectives of 
the subsequent motion controller. The first objective is to move the platform along 
the trajectory towards the desired pose and the second objective is to keep the plat- 
form on the trajectory. 








measured desired, max, 
pose s pose s acceleration 
a {starting velocity velocity| , 
Swe | pose s , ~~ desired pose le 
> -° along trajectory 
S deviation velocity | Vip with max 
from trajectory actuation time) 














m 
Sue 


Figure 3.5. Left: The trajectory controller yields the local deviation from a given trajectory 
and a local pose along that trajectory. Right: The motion controller calculates corresponding 
velocities and combines them to a desired velocity. 


First, an MUG pose s, on the line between the start pose s},, and the 
desired pose si. is calculated, such that the intermediate pose has the shortest 
distance to the measured pose s}, : 


s ere ce (3.1) 
sv = si Sio (3.2) 


u describes the relative pose of the intermediate pose between the start poses, . , 
where yu’ = 0, and the desired pose J , where x’ =1. The relative pose is calcu- 
lated individually as yw’ for the measured position and as w'” for the measured 
orientation by using only the first two components and the third component of the 
vectors in Equation 3.4, respectively. yw’ and u” are bounded to the range be- 
tween 0 and 1, resulting in intermediate poses that are located between start pose 


64 Helge Hülsen 


and desired pose. yu’ is then composed as a weighted sum of yu’ and u”, allow- 
ing for a dominance of translation or rotation during pose control. In the current 
implementation, when position and orientation are controlled simultaneously, the 
weights are r“’ = 0.7 and r“” = 0.3, such that position control dominates. 





u = r”! ue 4 phn ue f (3.3) 
s” s” l 

w = l > l ; (3.4) 
S sS 0 

S” = $0 T She: (3.5) 


The global deviation sf, from the trajectory is the difference between the 
measured pose and the intermediate pose, and the global pose s\,. along the tra- 


jectory is the difference between the intermediate pose and the desired pose: 


She = She = She d (3.6) 
d A 
Sre = Sic =se < (3.7) 


m 


The global poses s‘,, and s',, are shifted by the negative measured pose —s’, ., 
and their position vectors are rotated by the negative measured angle —¢/", to yield 
the corresponding local poses s;, ands,,. These two steps, translation and rota- 
tion, can be comfortably performed with homogeneous matrices [39]. Finally, the 
local Cartesian poses are transformed into local polar poses s; „ and s, „, by apply- 
ing the rules stated above. 


3.2.3 Motion Controller 


The motion controller (Figure 3.4 and Figure 3.5 — right side) calculates a desired 
velocity vi, such that the microrobot constantly decreases its velocity to zero at 
the desired pose. 

The correcting velocity v;,, and trajectory velocity v; p are calculated from 
the local deviation s;,, of the mobile platform from the trajectory and from the 
local pose along the trajectory, respectively: 


Vp S320" +8, yo (3.8) 
O, = 28° -Pp ; (3.9) 
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Vip HZ OS,» > (3.10) 
O, p = 2a" -Pp (3.11) 


The velocities are decreased with an acceleration a“ that is smaller than the lower 
bounds of the maximum acceleration a” . Analogously, the angular velocities are 
decreased with an angular acceleration a“ that is smaller than the lower bounds of 
the maximum angular acceleration @”" . The values for a” and a” are esti- 
mated beforehand. There are two uncertainties during closed-loop control that 
motivate this artificial damping. Firstly, the measured velocity v;", can be higher 
than the desired velocity vi , mainly due to errors in the actuator controller model. 
Secondly, the sensor sampling rate and thus the update rate of the desired velocity, 
has some variance. With an artificially smaller acceleration, the maximum allowed 
desired velocity is smaller and a certain jitter in the sensor sampling rate is allowed 
before an emergency stop is required (Section 3.2.5). 

The local velocity vectors v;, and v,, are then added to a desired local veloc- 
ity vector Vie , which is scaled down by the highest factor by which any of its 
components exceeds its maximum value. Depending on the implementation of the 
successive actuator controller, the output of the motion controller is given in polar 
coordinates vi p (for manually trained maps in Section 3.5.2) or in Cartesian coor- 
dinates vi (for automatically trained maps in Section 3.5.3). 


aX 


3.2.4 Actuator Controller 


The actuator controller (Figure 3.4) finally tries to achieve the desired velocity Vie 
by manipulating the parameters p° of the actuator signals correspondingly. The 
actuator controller is implemented as SOLIM as shown in Section 3.5. 


3.2.5 Flexible Timing During Pose Control 


For the control of microrobots in an SEM there are two constraints that have to be 
considered: 


1. The sampling rates of the sensors differ and vary considerably. This prop- 
erty is due to the dependency of the acquisition and processing time on im- 
age properties like image size, model size, noise, ete. Especially when 
processing SEM images, the acquisition and processing times can vary be- 
tween some milliseconds and some seconds. 

2. The actuation time has a lower bound T^, and an upper bound Tf such 
that the robot does not move "blindly" when sensor sampling periods are 
too long. 


Therefore, a controller update is triggered by the arrival of new sensor data. Then 
new actuation parameters are calculated by the low-level controller, the corre- 
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sponding signal is copied to the ring buffer of a multichannel D/A board, and the 
output of the signal is started or continued. 

In addition, two timers are set after the reception of a new sensor data to take 
into account the timing constraints. Firstly, a stop timer is set to T°, which stops 
the robot after the maximum actuation time. The latest stop time T° is calculated 
such that the microrobot does not move further than to the desired local pose along 
the trajectory. For a translation movement this means that after moving a time T° 
with a velocity (1+ r’ ve p » there must be enough time to break with the maximum 
acceleration a” : 

Sip ô Qr Wis 


max (l i ri b a”” 


a 





(8.12) 


The coefficient r” estimates the velocity error, i.e., if the desired velocity is 
v, the microrobot moves with a+r), in the worst case. T” is similarly 
calculated for rotation and the stop timer is started with the lower value. Secondly, 
a request timer is set to 7^, which requests the sensor server after the minimum 
actuation time to send new sensor data as soon as it has arrived. The minimum time 
T^„ can be used to limit the communication traffic or the update rate of the con- 
troller. The two timers are implemented with help of the Windows multimedia 
timers, since they provide the necessary accuracy of about | ms, assuming that 


external disturbances from interrupts, efc. are prevented. 


3.3 The SOLIM Approach 


The main structure of self-organizing locally interpolating map is based on the 
standard SOM approach, where input support vectors are associated with output 
support vectors. The application of the SOLIM approach as ideal inverse model 
controller then defines the main working principles. SOLIM mapping is an exten- 
sion of simplex interpolation as it is used in music synthesis control, and it also 
features extrapolation. SOLIM learning is divided into an approximation and a 
self-organization part. The approximation part locally learns the output support 
vector that has been most responsible for the measurement, and the new self- 
organization rule locally rearranges the neighboring vectors around this output 
support vector. 


3.3.1 Structure and Principle 


The task of the SOLIM network is to learn a continuous, smooth inverse model 
m:GwW P ofa system behavior s: Pt) G , such that the difference between the 
desired system output g’ and the measured system output g” is minimized 
(Figure 3.6a). 
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Figure 3.6. a. SOLIM learns an inverse model of the system behavior. b. Step 1: SOLIM 
mapping m: g’ > pf, step 2: system response s:p° > g”, step 3: approximation by 
updating P - © Step 4: self-organization in output space by arranging all neighbors of 
Apn- 

The structure of the SOLIM network is similar to the structure of extended 
SOM [18] (Figure 3.7). The network N consists of | N | nodes n, =(g,,p,,¢,,B,) - 
Each node n, associates an input support vector g,¢G with an output support 
vector p; € P and thus forms a kind of look-up table. In addition, each node n, 
has an associated support vector ¢, in topology space C and a set B, of nodes 
n;, that defines its neighborhood. 

The following steps are performed during operation in the context of a learn- 
ing controller, i.e., during the learning of a model of an inverse system behavior 
(Figure 3.6b, Figure 3.6c, and Figure 3.7). These steps are explained in more detail 
in the following sections. 


e Mapping. A desired system output is mapped to actuation parameters 
m:igi Wp’. 

e System response. The actuation parameters p“ are applied to the system 
and the system output is measured s:p“ > g”. 

e Approximation. The pair (g”,p°) describes the system behavior. Using 
g” as map input yields a winning node n „ that has the highest influence 
with respect to g”. Therefore, the corresponding output support vector 
P,» is updated such that the mapped output of g” is approximately p°. 
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e = Self-organization in output space. The output support vectors around the 
winning node n „ are updated with the help of a self-organization rule to 
arrange the output support vectors according to the given topology. 

e Self-organization in input space. The input support vectors can be pre- 
arranged according to the given topology or they can be learned with the 
help of a self-organization rule, where all input support vectors belonging 
to neighborhood of the winning node g” are moved towards g’. The 
winning node n_, is the node that has the highest influence with respect to 
the input vector g“. 


input space G topology space C output space P 
@ g” =s(p° ) © p“ =m(g" ) 
gge % g Ce e G P: @ r 
e— — ——_#’—9 e— --—e —9 e— ——<e <0 
Zue g, C C, Pue Py 
a 
input space G topology space C output space P 
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Figure 3.7. SOLIM operation with a. a 1D network and b. a 2D network. Step 1: SOLIM 
mapping m: g’ > p^, step 2: system response s:p“ }> g” , step 3: approximation by 
updating Pn » Step 4: self-organization in output space by arranging all neighbors of N n- 


3.3.2 Mapping 


The mapping part of the SOLIM algorithm framework finds a unique output vector 
p“ €P for each input vector g” € G and is solely dependent on the network of 
nodes, i.e., on the topology-supported association between input support vectors 
g, € G and output support vectors p, € P . The input vector g“ may lie within the 
convex hull of neighboring input support vectors such that the output vector is 
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determined as interpolation between the output support vectors. If the input vector 
lies outside the convex hull, the output vector is determined by extrapolation of 
the border nodes’ behavior. In cases where the dimension d“ of the topology 
space is smaller than the dimension d? of the input space (Figure 3.8), the separa- 
tion is not that clear, and an input vector can be eligible for interpolation and ex- 
trapolation. 


input space G topology space C output space P 
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Figure 3.8. Mapping from a 2D input space to a 1D output space with a 1D network 


The mapping is performed by first analyzing the input vector in input space and 
then synthesizing the output vector in output space. 


e Analysis. The geometrical relation of the input vector g” with respect to 
the arrangement of input support vectors g,, which is defined by their 
neighborhood B,, is analyzed, yielding an interpolation value f, and an 
extrapolation value x, for each node n, . 

e Synthesis. The output vector p° is synthesized, such that its geometrical 
relation with respect to the arrangement of output support vectors p, corre- 


sponds to the interpolation values f, and to the extrapolation values x, . 


i 


By using the same characteristic influence values f, and x, during the analysis 
and during the synthesis, p° has the same geometric relation to the arrangement of 
output support vectors as g” has to the arrangement of input support vectors. 

Before applying the output of the mapping to a real system, it is bounded by 
minimum and maximum values to prevent the system from being damaged by 
excessive voltages or currents. These values must be known beforehand and are 
parameters during the initialization phase. 


3.3.2.1 Interpolation 

SOLIM interpolation is based on simplex interpolation as it is used in music 
synthesis control [40]. A simplex is the d“ D analog of a line in 1D, a triangle in 
2D and a tetrahedron in 3D. Generally, a d“ D simplex is defined as the convex 
hull of d] +1 nodes, often called vertices. In the scope of SOMs, the arrangement 
of topology support vectors defines simplices, which correspond to simplices in 
input and output space. During simplex interpolation, the relative pose of an input 
vector g” within a simplex in input space is used to determine an output vector p° 
with a corresponding relation to the associated simplex. Simplex interpolation has 
been chosen as the basis for development because it is a local technique, it is 
continuous, and it is relatively simple [41]. On the other hand, simplex 
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interpolation requires that any input vector appears in the convex hull of a simplex. 
This implies that topology and input space have the same dimension. In many 
robotic applications, however, it is more desirable to have a network with a 
topology space dimension d“ that is smaller than the input space 
dimension df, e.g., when a tool centre point is monitored in 3D by two cameras. 
4D input support vectors that are arranged in a 3D topology would be the most 
efficient model. 

SOLIM interpolation is based on simplex interpolation but changes the focus 
from simplices to nodes because with d° <d? an input vector g” will probably 
not be within a certain simplex defined by input support vectors, but close to the 
network of input support vectors e.g. a 2D input vector will probably not be exactly 
on one of the lines defined by neighboring input support vectors that are arranged 
in a 1D topology, but it will only be close to one or more lines (Figure 3.8). There- 
fore, instead of interpolating between the nodes of one simplex, each node has a 
continuous influence with respect to the current input vector. The influences are 
then normalized and used as weights for the corresponding output support vectors 
to generate the output vector. 

During the analysis part of the mapping (see above), an influence f, is calcu- 
lated for each node n,. This influence is maximal for input vectors g“ that are 
located at the corresponding input support vector g, and decreases to 0 at limits 
that are solely defined by the associated set B, of neighboring nodes (Figure 3.9). 
In the input space G, each limit /, , € L, can be defined by a position vector gi P 
on the limit and a normal vector g;, perpendicular to the limit. The derivation of 
gi, and g;, from the set of neighboring nodes will be given later in this section. 

The influence of a node n, with respect to an input vector g” depends on the 
relative distances d, ,, j=1,2,...| L, | of the input support vector g, to the input 
vector g” towards the limits l, , (Equation 3.13). This relative distance measure is 
related to differences between support vectors and takes into account the direction 
g;, towards each limit, such that it implements a low-level normalization with 
respect to the network of support vectors. An explicit normalization is therefore not 
necessary. 


2 (o4_o 
d,; -ilee (3.13) 
gi (s-s ) 














A blending function then maps the relative distance d,, between input vector and 
input support vector to the influence f,, of the node n, with respect to the limit 
L 


ij’ 


f =(d,,). (3.14) 
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Figure 3.9. Influence f; of a node n, a. defined by two neighbors in a 1D network and b. 


i 


defined by six neighbors in a 2D network. f, is 100% for g’ =g, and decreases to 0% at 


i 


the influence limits. 


The blending function yields an influence of f;,=1 for d,, <0, an influence of 
J; =9 for d; ,>1, and a linear transition in-between: 


b(d,,)=1-d,, for 0<d,, <1. (3.15) 


When the input vector g” lies within the simplex defined by the input support 
vector g, and the neighboring input support vectors defining the influence limit 
l, ,, each influence f; ; is identical to the barycentric coordinates of g’. 

The influences f,, of a node n, with respect to each limit /;, are then com- 
bined with the min(-) operator to yield the influence f, of the node (Equation 
3.16). The min(-) operator selects the highest of the limit-specific influences f, , 
and thus selects the most relevant of the set of simplices that are defined by n, and 
all of its neighbors (Figure 3.9). For the continuity of the whole mapping it is im- 


portant that the min(-) operator is a continuous operator: 


f, = min( f) (3.16) 
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In the synthesis part of the mapping, the influence values f., which describe the 
relative closeness of the input vector g” to the input support vectors g,, are used 
as weights to combine the output support vectors p, (Equation 3.17). Therefore, 
the output vector has the same relation to the arrangement of output support vec- 
tors as the input vector has to the arrangement of input support vectors. Both rela- 
tions are expressed by the influence values f, : 


P=) fP, - (3.17) 


Í, are the | N| influence values f, but normalized, such that their sum pay 
becomes 1. In standard simplex interpolation, the sum of influences f, is always 1. 
In SOLIM interpolation, however, the sum of influences f, can have any positive 
value because the input vector does not necessarily fall within exactly one simplex 
and because nonlinear blending functions might be used. Therefore, downscaling is 
performed if the sum £ J, ofall | N| neurons’ influence values exceeds 1. If the 
sum £ f, is smaller than or equal to 1, downscaling is not performed to avoid 
division-by-zero problems for the case when all f, = 0 . Instead, all influence val- 
ues are increased uniformly such that the sum becomes 1: 


f i 
i For 
TE ÈA à (3.18) 


f,+(1- > A )/IN| otherwise. 


3.3.2.2 Influence Limits 
SOLIM interpolation relies on the definition of an influence f, for each node that 
depends on the current input vector g” and that defines the weight for the synthe- 
sis of the output vector p°. The influence for the node n, is f, =1 for an input 
vector g” =g, and decreases to f, = 0 at its limits. 

There are two groups of nodes that influence the definition of a limit /, , for 
a node n, (Figure 3.10). Firstly, the limit is defined by the neighboring nodes 
Hi im , m=1,2,...d° , that, together with n, , define the simplex s,,- This definition 
allows SOLIM interpolation to be similar to simplex interpolation because the 
influence of the node n, is f, =0 for input vectors that are located at the facet that 
is opposite to g, of the simplex in input space. This means that for an input vector 
g” that is located at any input support vector g, only the influence of the corre- 
sponding node n, is f, #0 and the output vector is exactly the corresponding 
output support vector p° =p,;. The definition of the limit /,, with ee is suffi- 
cient for the case when the input space dimension d° is equal to the topology 
space dimension d°. In that case, d° nodes n;,„ define all of the limit’s d° 
degrees of freedom. When the input space dimension d° is larger than the topol- 
ogy space dimensions d°, the neighboring nodes n, ,„ only define d° of the 


m 
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limit’s d? degrees of freedom. The remaining degrees of freedom are defined by 
the node n, itself and the node n;, that shares the facet given by Nym ; 
m=1,2,...d° of the simplex. This definition allows having the same limit defini- 


tion for n, and for n. 
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Figure 3.10. Limit of node n, is defined by other nodes Nja ; n, yash ro of simplex 
S,,;> by node n, and by node n; on other side of simplex facet. a. 1D topology, line 
simplices. Limit normal defined by g, and g; ; b. 2D topology, triangle simplices. Limit 


normal defined by g, and gjo and g, 4,23 © 3D topology, tetrahedron simplices. Limit 
normal defined by g, and g; ,, and g; ,, and gi ,,. 


During SOLIM interpolation the limit /, , is defined by the position vector gi, on 
the limit and the normal vector g;,, which is perpendicular to the limit (Figure 
3.10). The position vector can be found as the input support vector corresponding 
to one of the neighboring nodes n, jm belonging to the simplex s; . 


Z: =~ Zija (3.19) 


The calculation of the normal vector is started with the sum of the normalized 
differences g” = g; 178g; and g =g 7 -g; jı» Which involve the node n, itself 


and the node n, ; on the other side of the limit: 
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gij z a E i (3.20) 


This definition leads to equal angles betwsen the limit normal g;, and g” and 
between the limit normal g;, and g°. As an example, assume that g; is a 3D 
support vector belonging to a network with 1D topology space (Figure 3.10a). The 
position vector of one of its two limits /,, would be defined by its neighbor g,,, 
and the normal vector would be defined iby the support vector g, itself and the 
support vector g,,, on the other side of the limit. There are cases when there is no 
node g;, on the other side of the facet which is opposite to n, . Then the g” -part 
of Equation 3.20 vanishes. 

In the case of a 1D topology (d| =1), the definition in Equation 3.20 is suffi- 
cient. In all other cases (d° >1), the limit normal g; , must be perpendicular to the 
differences gi ,,, k=2,3,.. .d° between the support vectors defining the limit 
facet opposite to g: 























ie ge Bij (3.21) 
g3 E83 T Sij (3.22) 
ean (3.23) 
g a EE ae Bias (3.24) 


An iterative algorithm then makes the limit normal g; , as defined in Equation 3.20 
perpendicular to all g; ,,. In each step the difference vectors g; ,, that have not 
been applied on g;, yet ate also made perpendicular to the current difference vec- 


tor g; ,, to make the changes on g; ,, being linearly independent of each other. 


Make g;, and g; ,,,. ea qc perpendicular to g; if 
Make g;, and g; ,,... +8), _, perpendicular to g; , , 


e 
e 
e 
° Make gi , and g'e A ae to gi, 

e Make g;, perpendicular to g, 

3.3.2.3 Extrapolation 

SOLIM extrapolation adds an extrapolation component př to each output sup- 
port vector p, before combining them into an output vector: 


P =x,-P’, (3.25) 
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IN| 
p=) jp (P +p). (3.26) 


The weight x, for the extrapolation component p; defines the distance of the 
input vector g” from the input support vector g, in the direction of g7 , as shown 
in Figure 3.11 and Figure 3.12: 


d_ 1 
x, = Eea (3.27) 


gi 


g’ is the mean difference vector between the input support vector g, and all input 
support vectors gf , in an extrapolation neighborhood B/ . Analogously, p” is the 
mean difference vector between the output support vector p, and all output support 
vectors in the same extrapolation neighborhood B? : 


g= > (2-24), (3.28) 
nf EBP 

p?= > (p-p). (3.29) 
nf, eB 


This neighborhood B’ is the set of the 3% —1 direct neighbors in the d| D topol- 
ogy space C , i.e. the two direct neighbors for d° =1 , the eight direct neighbors for 
d° =2, etc. 
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Figure 3.11. Extrapolation component př =x,-p” corresponds to relative distance 
g? =x,-g” of input vector g“ from border input support vector g, 
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Figure 3.12. Extrapolation weight x;: Distance between g’ and g, in relation to mean 
distance g” between g, and its neighbors. 


3.3.3 Learning 


There are two goals for the learning task: SOLIM mapping approximates an in- 
verse system behavior, and the input and output support vectors are both arranged 
according to a predefined topology, such that similar input vectors are mapped to 
similar output vectors and the mapping thus becomes smooth. 

In standard SOM learning, approximation and self-organization in output space 
are performed in one step by using the estimate for approximation as attractor for 
the self-organization of the output support vectors. In PSOM learning, the com- 
bined approximation and self-organization is even performed in a combined input- 
output space. In SOLIM learning, the single steps are separated to better control 
their influence on the map. The approximation step updates the output support 
vector of the node n „ that has the highest influence on the pair of system input 
p° and measured system output g” . The self-organization in output space then 
locally arranges the output support vectors of the nodes around n „ according to 
the topology and thus ensures that the inverse of the mapping is continuous as well. 
The self-organization in input space, if activated at all, arranges the input support 
vectors of the nodes according to the topology. 


3.3.3.1 Approximation 
The idea behind SOLIM approximation (Figure 3.13a) is to change an output sup- 
port vector p „ in the mapping p” = m(g” | , such that p° =p”, i.e., the SOLIM 
map passes through the point (p’.g”| of the system behavior. There are several 
output support vectors that have an influence on p” but since only one point of the 
system behavior is given, the output support vector p „ with the highest influence 
is used for updating. 

First, the synthesis equation for p” is rearranged to identify the influences of 
different output support vectors on the output: 


m 


p° =p", (3.30) 
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D "Pit J Pi > 3.31) 


with f 4 and x” being the interpolation and extrapolation weight, respectively, of 
each node n, with respect to the input vector g”. é” is just a substitution for 
1+x” |B? |, where | B’ | is the number of neighbors of node n, contributing to 
the extrapolation component p?. p” is the sum of the output support vectors of all 
these extrapolation-relevant neighbors. Equation 3.31 shows that the support vector 
p yn of the most relevant node n „ appears in the first sum of p° and as part of 
p”. The second appearance can be neglected because P, is only a neighbor of a 
few border nodes with a low influence F, < Í a 


SOLIM system SOLIM system 
behaviour behaviour behaviour behaviour 
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Figure 3.13. Approximation: a. Preliminary estimate P,» by inversion of mapping. b. 
Better estimate p°? by scaling down support vector change P. — Pi»: 


Solving Equation 3.31 for p „ yields an expression for a preliminary esti- 
mate p,» of p_,,, and calculating the difference between the estimate p,» and 
the old value of p „ , yields a relatively simple expression: 


: 1 
P -Pp == 
F Sp 





(p° -p”). (3.32) 


The substitution č „ is given by the simpler notation ¢ „ =¢"_,, . The interpola- 
tion and extrapolation influences of the winning node n „ follow the same rule 
with ae = =f, _n and x ,, =X" a , respectively. 

If p,,, were to be updated to P,a, g” would be mapped to p°. But, assuming 
that the gradient of the SOLIM map is similar to the gradient of the system behav- 
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ior (Figure 3.13b), scaling down the change Ĥ -p „ of p, by Faj yields a 
better estimate p°“ for p „. The interpolation weight Ír is chosen to scale 
down because it describes the ratio of p° -p „ to Py» — P» : 


p° =P. + F lP, P,e > (3.33) 
p“ =p, +e" =p”). (3.34) 


e,a 


Finally, p,,, is moved towards p 
EES 


with a constant approximation learning rate 


pie? = pin? +(pe pi?) (3.35) 


3.3.3.2 Self-organization in Output Space 

During SOLIM self-organization, all neighbor pairs (n,,n,)€ R of the winning 
node n „, whose topology vectors are in-line with respect to c „ in topology 
space, are changed in the output space to be in-line with p_,,. Each of these 
neighbor pairs consists of a reference node n, =(g,,p,,¢,,B,), which has the 
smaller approximation error A, and remains unchanged, and a candidate node 
n,= (g..P..¢.,B.) , which has the larger approximation error A, and is updated 
with a self-organization learning rate ¢?* (Figure 3.14): 


(new) _ 


pi" = pe 4 er Ps (pe -p). (3.36) 


For each candidate node n, an estimate for its output support vector p, is found 
by aligong p. to the difference (P> - » —P, | and by adopting the relation between 
ie | and le, » ~%,|| in the output space (Figure 3.15) 


























p =p, + "Pp, (3.37) 








E g, 


The self-organization learning rate is calculated for each candidate node and 
depends on the approximation learning rate £”°, on a ratio r” and on the candi- 
date node’s number of neighbor pairs u, =| B? | /2: 


er =1—(1-r? sh)", (3.38) 
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Figure 3.14. Self-organization in output space by aligning a candidate support vector p, 
with respect to the winning support vector P „ and the opposite reference support vector 
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Figure 3.15. a. The alignment by a self-organization step in 1D. b. One of four alignments 
during a 2D self-organization step. 
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Averaged over time, an output support vector is moved u, times in a self- 
organization step when it is moved once in an approximation step (Figure 3.16). 
Thus an output support vector is moved relatively with d?* = 1-(1-e?" “ to- 
wards a self-organization estimate p° when it is moved relatively with 


d?’“ =e?” towards an approximation estimate p°“ : 


1-(1-e?*)" 
r’ = pa (3.39) 
approximation self-organization 
o o x, 
P.r p“ P | p” 
e" l-e” 1-(-e”) (-e") 
d” d” 


Figure 3.16. Left: Relative change d”? of p „ towards its estimate p°“ after one ap- 
proximation step with update rate £”. Right: Relative change d” of candidate output 
support vector p, towards its estimate p^“ after u self-organization steps with update rate 
ee 


Solving Equation 3.39 for ¢?" then yields Equation 3.38. The ratio r” is used 
as a parameter of the self-organization learning rate in Equation 3.38 and depends 
on the degree of disorder 0 of the three nodes n,, n,, and n,, and on the ap- 
proximation error A, of the candidate node n, : 


p Soman i AANA yh AL) a a E gOS), (3.40) 


where y= t(x, x", y',y*,x) describes a linear transition from y’ to y" between 
x' and x". Figure 3.17 shows the characteristic curve of r” with typical values of 
Al =0.1, Ar =1.0, 7? =0, 7?" =0.7, 0° 330°; 6" =120°, r°?” =0, and 
r?*" =1.0. The idea behind Equation 3.40 is to decrease the self-organization 
learning rate smoothly from a high value for disordered SOLIM maps that ap- 
proximate badly, to a low value for ordered SOLIM maps that approximate well 
(Figure 3.18). 
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Figure 3.17. Characteristic of relative change r” depends on candidate node’s approxima- 
tion error A, and on degree of disorder Ô of n, , N n and n, 
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Figure 3.18. Different possible cases and possible result of self-organization (dashed ar- 
rows). a. System behavior g“", g^? <[0.0,1.0]. Local disorder leads to locally high ap- 
proximation error. b. System behavior g” = pf + p} . Low approximation error but not well- 
ordered output support vectors. c. System behavior g“", g“” e [-0.5, 1.5] . Ordered output 
support vectors but locally high approximation error. d. In all cases, self-organization helps 
by decreasing the degree of disorder and the approximation error. 
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SOLIM self-organization minimizes the angle between p,—p,,, and P —p,, 
which is consequently chosen as a measure for the degree of disorder 6 . A simi- 
lar "degree of regularity" is defined in [42]: 





a= arccos(d,, «d,,) oo 3 (3.41) 
T 
dpa eab po eo (3.42) 
P.-P,» |p -P. 




















The approximation error is measured and stored for each node. The difference 
between c” and c”, which are the corresponding mapping images of g” and g@ 
in topology space, is stored as the approximation error A, of the node n, : 


A, =|e*-c"|. (3.43) 


ce’ and c” are computed analogously to p° and p”, respectively. g’ is computed 
analogously to d? (compare Section 3.0): 


cf =. £ Filete) : (3.44) 
i=1.-|N| 

= > fi (e+e) . (3.45) 
i=1N] 


Measuring the approximation error in topology space makes the measure relatively 
independent of scale differences between the dimensions and relates it to the 
smallest distance between topology support vectors. 


3.3.3.3 Self-organization in Input Space 

Self-organization in the input space G is often not necessary because in typical 
robotic applications the input support vectors can be predefined according to a 
topology, e.g., to cover the desired velocity range of the mobile microrobot plat- 
form, the input support vectors can be placed on a 3D grid. If this is not the case, 
they can be learned with the Kohonen self-organization rule [18] with the input 
vectors g as attractors. Special attention must be given when succeeding input 
vectors g” are strongly correlated. This is the case in many robotic applications if 
the tool centre point moves along a trajectory [43, 44]. The classic Kohonen self- 
organization rule will then let the input support vector network contract along the 
g“ -trajectory if the learning rate is not small enough. In addition, when changing 
the input support vectors g, the corresponding output support vectors p, must be 
changed accordingly. Throughout this work, the input support vectors are prede- 
fined and kept constant. 
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3.3.4 Conclusions 


The SOLIM approach has been designed to implement a learning inverse model 
controller. 

The SOLIM structure is inherited from SOM, where input support vectors are 
associated with output support vectors and topology support vectors, together 
called nodes. In contrast to SOM, however, the neighborhood relation between two 
nodes must be given explicitly during a topology construction, and it also defines 
simplicial complexes in input space, in output space and topology space. 

SOLIM mapping consists of an interpolation and an extrapolation component. 
Both components are based on transferring the position of a given input vector 
with respect to the input simplicial complex to an output vector having a similar 
position with respect to the output simplicial complex. The mapping is continuous 
because the dependency of the influence of each node on the input vector is con- 
tinuous. Moreover, the influence of each node is limited by the input support vec- 
tors of its neighbors, such that the mapping is local and thus interpretable. 

SOLIM learning also follows a different approach compared to existing SOM- 
based controllers. Firstly, the approximation and the self-organization phase are 
separated to better adjust their properties. For the derivation of the relatively sim- 
ple approximation update rule, the mapping from a measured system output to the 
applied system input is solved for the output support vector with the highest influ- 
ence on the measured system output (winning node). During self-organization, all 
neighbors of the winning node that are opposite in topology space are aligned to 
become opposite in output space, as well. The new learning approach is local be- 
cause only the output support vector with the highest influence and its neighbors 
are updated during each iteration step. As a second prerequisite for incremental 
learning, all learning parameters are kept constant or are adapted automatically. 
Finally, the self-organization rule not only depends on the degree of disorder but 
also on the approximation error, such that inverse model ambiguities are resolved 
towards an ordered output support vector arrangement with a low approximation 
error. 


3.4 SOLIM in Simulations 


This section assesses the mapping performance and the learning performance 
with respect to systems with known behavior. The systems are modeled as func- 
tions p° }» g” with different characteristics. 


3.4.1 Mapping 
The first important feature of the SOLIM framework, the mapping performance, is 


evaluated by initializing the SOLIM map with known support vectors and by com- 
paring the output with the output of the function the map approximates. A typical 
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system for approximation tasks where numbers on approximation errors are avail- 
able is the 2D Gaussian bell, which is defined as 


Cern uT 
p“ = gauss(g“) exp All 7 J 2 ; (3.46) 


In the following simulations, the mean and the standard deviation will be set to 
u=0.5 and o =0.1, respectively (Figure 3.19a). The mapping is initialized with 
5 x 5 nodes, whose input support vectors g, are placed at the positions 
Zi Z2 € {0.1,0.3,0.5,0.7,0.9} and whose output support vectors p, are placed at 
the corresponding positions p, = gauss(g,) . 

To evaluate the interpolation performance, the RMS error for the SOLIM 
map is computed in the range g”, g”? e [0,1] (Figure 3.19b). The RMS error of 
0.043 is comparable to the RMS errors of other algorithms, which are listed in 
Table 3.2. To evaluate the extrapolation performance, the RMS-error is com- 
puted in a larger range g%', g”? € [-0.5,1.5] (Figure 3.19c). The extrapolation is 
reasonable, but the RMS error of 0.057 is relatively large because the falling slope 
at the borders is continued, although the Gaussian bell approaches the zero plane. 
This error can be decreased significantly by using more neurons that better repre- 
sent the slopes, e.g., by placing 7 x 7 neurons in the same area, yielding an RMS 
error of 0.024 (Figure 3.19d). 








Table 3.2. RMS error for approximation of 2D Gaussian bell (42 = 0.5, o = 0.1) with 25 
support vectors in the range go, g” E€ [0,1]. All values from [38], except for SOLIM, 
which is based on simulation of 51 x 51 points within same range (see also [45]). 








Algorithm RMS error 
PSOM 0.79 
local PSOM 0.049 
SOLIM 0.043 
RBF 0.041 
I-SOM 0.040 


Cp -SOM 0.016 
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Figure 3.19. a. 2D Gaussian bell, g^', g“” e€ [0.0,1.0]. SOLIM approximation with b. 5 x 


5 nodes, go, g” € 0.0,1.0] ,c. 5 x 5 nodes, aan g” € [-0.5,1.5] and d. 7 x 7 nodes, 
g”, g*?in[-0.5,1.5]. 


3.4.2 Learning 


The second important feature of the SOLIM framework, the learning performance, 
is evaluated by applying the SOLIM map output to a virtual system and by using 
the measured system output to learn the SOLIM map. For a virtual two-joint kine- 
matic module with a changing behavior, an inverse map is learned incrementally at 
different similarity coefficients for successive desired system outputs. 


3.4.2.1 Procedure 


For the evaluation of the learning performance of the SOLIM approach, the same 
procedure is used for all simulations. It consists of an initialization phase, where 
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the network is set up, and the actual operation phase, where the network controls 
the system and is continuously updated. 
For the initialization phase only three specifications are necessary: 


l. 


The size of the network defines the number of nodes in each topology di- 
mension and with it also the placement of the topology support vectors c, 
in topology space C. 

The input range defines the range for allowed desired input vectors g” 
and, if the input space dimension d° is equal to the topology space dimen- 
sion d“, it can also be used for initializing the input support vectors g, in 
an ordered arrangement. If d° #d°, e.g., a 1D topology in a 2D input 
space, the input support vectors are prearranged manually or they are ar- 
ranged during the operation phase by self-organization. 

The output range defines the range for the initial random arrangement of 
the output support vectors p, and the boundary for the output vectors p°. 
If knowledge on an inverse model already exists, it can be used to initialize 
the output support vectors and thus to provide a learning bias. 


It is not necessary to initialize the learning parameters differently for each system 
behavior. During the simulations (and during all experiments of the next section) 
the parameters have been set to the values given in Table 3.3. These values have 
been found by alternately analyzing the influence of a quantity and selecting the 
best of the resulting new combinations. This procedure is similar to genetic algo- 
rithms, and its automation could reveal parameters that show a slightly better learn- 
ing performance. 

The operation phase consists of repeating the combination of mapping and 
learning, which is called incremental learning or online learning [33]. In more 
detail, the steps during operation are as follows (Figure 3.20): 


A desired system output g” is generated randomly. 

SOLIM maps from the desired system output g” to a system input p°. 
The system input p° is applied. 

The system output g” is measured. 

SOLIM training is performed with the 3-tuple (g’.p’, g”) : 


simulated 
system output 


m 


desired 
system output 





system input 
p 












system 





controller 


Figure 3.20. Simulation setup: The controller learns a direct inverse model g’ > p° of the 
system behavior p“ > g” by using the simulated system output g” . 
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Table 3.3. Learning parameters that are used during all simulations and experiments 














Parameter Value 
learning rate for approximation in output space sP! =0.8 
lower bound for approximation error Al =0.1 
upper bound for approximation error A? =1.0 
lower bound for relative SO-change pr! =0.0 
upper bound for relative SO-change re^ =0.7 
lower bound for degree of disorder 8! =30° 
upper bound for degree of disorder 6" =120° 
lower bound for relative SO-change rêl = 0.0 
upper bound for relative SO-change prot 10 
threshold for deadlock counter A" =10 
scaling of "allowed subspace" A for new random Pa p=2.0 





3.4.2.2 Inverse Kinematics 
The SOLIM algorithm framework has been designed to be able to learn during 
operation and to adapt the inverse model to a system behavior that slowly changes 
over time. Furthermore, since all learning steps are performed locally, operation 
with correlated input vectors does not harm parts of the map that already have a 
low approximation error. 

A model of a two-joint serial kinematic (Figure 3.21a) is used as system be- 
havior. The model defines the mapping from the joint angle vector p° to the global 
position vector g” ; 


grh -cos( p")+1, -cos( p” + p”), (3.47) 


g” = sin(p”)+,-sin( p” + p°). (3.48) 


The axes’ lengths are / =10 and /, =5 until iteration step 3000 and /, =12 and 
l, =6 after iteration step 5000. Between these two phases there is a linear transi- 
tion of /, and /,, i.e., between iteration step 3000 and 5000 there is a slow change 
of the system behavior. The tool centre point (TCP) at the end of the second axis 
will reach positions in the range g”",g”” e [4.5, 10.6] , which are always reachable 
before, during, and after the system behavior's change (Figure 3.21b). Figure 3.21c 
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Figure 3.21. a. Configuration of a two-joint serial kinematic. b. 2D input space. Possible 2D 
output space c. before and d. after changing system behavior. Error development lg” -= g’ | 
with similarity coefficient e. 0.0 and f. 0.9. 


and Figure 3.21d show possible output support vector arrangements for the case 
1, =10 and /, =5 and for the case /, =12 and /, = 6, respectively. 

The SOLIM algorithm framework is able to adapt to changing system behavior 
during operation and even while the TCP is following given trajectories. "Follow- 
ing a trajectory" means that successive input vectors g” are similar to a certain 
extent c’, e.g., a similarity coefficient of c* =0.9 means that an input vector is 
chosen randomly around the preceding input vector within a range of +10% of the 
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whole range. For the first 2000 iterations, the similarity coefficient is c° =0 to 
allow the network to reach a stable state. This training phase is necessary since a 
map with random output support vectors cannot perform any useful operation. 
After this phase, the simulation is run with a similarity coefficient c° =0.9. The 
corresponding error developments for c° =0.0 and c* =0.9 are shown in Figure 
3.21e and Figure 3.21f, respectively. In both cases a relative error of about 5-107 

is reached until iteration step v = 3000. In Figure 3.21f the training phase was not 
completed until v = 2000 , but an error of about 5-10% could also be reached until 
v = 3000, even with a similarity coefficient of c* = 0.9 . In both cases, the relative 
error is constantly at a higher mean level at about 10° between v=3000 and 
v = 5000, but the error variation is higher for higher similarity coefficients. In both 
cases the relative error drops to about 5-10 within a certain period of time, but 
this period increases with the similarity coefficient. The effect of the similarity 
coefficient on the error variation and the settling time can be explained by the time 
the input vectors g” need before they have covered the whole input space. Because 
input vectors with a high similarity coefficient tend to "stay" in a limited region of 
the input space, the local learning rules only update the corresponding region of the 
map, such that relatively large parts of the map are untouched and keep their in- 
creased approximation error. The learning performance of the SOLIM approach 
thus depends on the relation between the speed of the system change and the speed 
of covering great parts of the input space by input vectors. 


3.5 SOLIM as Actuator Controller 


The SOLIM framework has been used to implement actuator controllers for differ- 
ent mobile microrobot platforms presented in Chapter 2. In this section, all meas- 
urements are performed with the mobile platform that is referred to as the effector 
platform. The same experiments with the stage platform have led to similar results. 


3.5.1 Actuation Control 


To make the robot move with a desired 3D velocity, typically the amplitude or the 
frequency of a sawtooth-shaped driving signal is modulated. In principle, the 
signal form can be modulated as well, but this approach would require some addi- 
tional research. Amplitude modulation and frequency modulation have different 
application areas, as can be seen from their effect on the velocity of the effector 
platform in Figure 3.22a and c, and Figure 3.22b and d, respectively. At low volt- 
age amplitudes, the velocity remains almost zero. Only above a certain threshold 
does the velocity increase fairly linearly with the amplitude. With increasing fre- 
quency, the velocity also increases linearly but the relation shows some nonlineari- 
ties and a lower repeatability at higher frequencies. The frequency can therefore be 
modulated to yield velocities in the lower range, while the amplitude can be modu- 
lated to yield velocities in the middle and higher range with a good repeatability. 
The threshold between the modulation methods can be found by relating a desired 
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velocity vy p and a desired angular velocity of, to the corresponding minimum 
values for amplitude modulation: 


d 








v 
drek __ Lp 
Vo =a (3.49) 
Vip 
d 
(a) 
d,rel _ lp 
Op = d,min ? (3.50) 
Lp 


and by taking the RMS value of these desired relative velocities 


ae (viv) + (obey. (3.51) 


Ip 


For values r” <1 frequency modulation is preferable, for values r^ >1 amplitude 
modulation is preferable. 

The frequencies of all actuation signals are set to be identical because they are 
generated simultaneously with the same multi channel DA board. With this con- 
straint, the number of DoF that can be controlled by frequency modulation is lim- 
ited to one such that the amplitudes must be modulated at small velocities to adjust 
the remaining two of three DoF. The current implementation of the motion control- 
ler uses the coefficient r” defined above to scale down the step frequency and to 
keep the desired velocity and thus the resulting voltage amplitudes at a minimum 
level when the effective desired velocity becomes small (r^ <1). Otherwise, am- 
plitude modulation is used. 


r! > min(r’,1), (3.52) 
faa O hes (3.53) 
Vip s (3.54) 
ao, alr’. (3.55) 


The changes in Equations 3.52 to 3.55 are optional, but have shown to significantly 
decrease the achievable closed-loop position error. During training, however, only 
the amplitude is modulated because it is easier to implement and also yields satis- 
factory results. Frequency and signal form are kept constant. 
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Figure 3.22. a. Influence of the signal amplitude on the platform’s velocity 
(w =u =75V, Jap = 700 Hz ). b. Influence of the signal frequency on the platform’s 
velocity (u° =u' =S50V, Jap = 1400 Hz ). c. Influence of the signal amplitude on the 
platform’s angular velocity (u° =u'=u°=75V, f „ =1400Hz). d. Influence of the 


step 
signal frequency on the platform’s angular velocity (u° =u* =u° =50V, f. p = 1400 Hz). 


ste] 


Solid line: mean, dotted line: standard deviation of 10 cycles. 25 amplitude or frequency 
increments. 1000 motion steps per measurement. 


3.5.2 Manual Training 


As a first approach, knowledge from experiments is used to manually implement 
an actuator controller for the microrobot platform. The actuator controller im- 
plements a mapping from a local, polar 3D velocity vector vi p toa 6D amplitude 
vector u’: 


F T 
d d a al a6 
v= i, of, | >u =|u Ses «Uh | (3.56) 
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by mapping the translation velocity by a SOLIM map and the angular velocity by a 
linear map: 


[vi AN hu”, (3.57) 
a, ue, (3.58) 
u’ = (u +u”). (3.59) 


The factor r scales down the combined amplitude vector u° such that none of its 
components is larger than the largest component of u®™ or u®” . 

The manual training of the SOLIM map for translation is a tedious task 
where a person tries to find suitable amplitude vectors u*” for 36 desired velocity 
vectors. In the case of the effector platform, the desired velocity vectors are all 
combinations of the velocities v, e {0.0,1.5,2.5,3.5} , given in mm/s, and the 
directions t, € {-180,-135, —90,—45, 0, 45, 90,135,180} , given in degrees. The 
desired velocity vectors are the input support vectors of the SOLIM map, and the 
amplitude vectors are the corresponding output support vectors. The SOLIM map 
is initialized as a 4 x 9 network, with the corresponding found input and output 
support vectors. In Figure 3.23a and b the input space and the resulting dependency 
of the first amplitude u“ on the input vector is shown. Extrapolation is turned off 
in Figure 3.23b to avoid a problem, which appears when input space dimensions 
are very different. In this case, the scale difference between 3.5 mm/s in the first 
dimension and 360° in the second dimension is very high. The solution to this 
problem is either turning off extrapolation where it is not needed, changing the 
dimensions’ scales to become similar, or finding a better extrapolation approach. 
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Figure 3.23. Effector platform: a. Input space covering absolute value and direction of 
translation velocity. b. Dependency of first actuation signal amplitude on desired translation 
velocity. Extrapolation is turned off. 
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The linear map for rotation is found by applying voltage signals to make the 
mobile platform rotate and by approximating the ratio of applied amplitude to 
measured angular velocity. The relation between voltage vector and angular veloc- 
ity is approximated as 


u” =o, [1.8 -5.5 18 -5.5 1.8 -5.5 (3.60) 


3.5.3 Automatic Training 


The SOLIM framework is used to learn the mapping from desired local Cartesian 
velocities Vie in three DoF to the actuation signal amplitudes u° in six DoF. 

During the initialization phase of the 3D-network, the 5 x 5 x 5 topology is 
associated with ordered input support vectors and random output support vectors 
within their predefined ranges. There is no more information required for the ini- 
tialization and operation of SOLIM since all learning rates are fixed or adapted 
according to the network state. During all experiments the learning parameters as 
given in Table 3.2 are used. 

Similar to the simulation procedure, the following steps are performed repeat- 
edly (Figure 3.24) during the operation phase: 


e A desired velocity vector vý, is generated randomly. 
e SOLIM maps from the desired velocity g“ = =v}, to actuation parameters 


a 


p. 
e The actuation parameters p° =u" are applied to the microrobot for some 


time. 
The velocity v;", is measured with help of a camera and image processing. 
SOLIM is trained with the 3- tuple (g p’.g ee 



















desired actuation 
local velocity parameters 
d d a a 
8 =V actuator p=u 
controller 
SOLIM 


l in 
measured local velocity g" = V; 


Figure 3.24. Experimental learning setup: The controller learns a direct inverse model 
g’ > p° of the system behavior p° + g” by using the measured system output g” . 


During the initialization of the SOLIM network, the input support vectors g, of 
the 125 nodes are arranged equally spaced within the range of the velocity space, 
while the output support vectors p, are randomly initialized within the allowed 
range for the amplitudes u“'*. With amplitudes in the range of [-80 V, 80 V] the 
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effector platform can cover a velocity range of xf., oe E€ [-1 mm/ s, 1 mm/s] and an 
angular velocity range of of, € [-39s, 3°/s| i 

The training of the effector platform is performed with uncorrelated input 
vectors g“ that are uniformly chosen from the input range. The input space and the 
first three dimensions of the output space after 2000 iteration steps are shown in 
Figure 3.25a and Figure 3.25b, respectively. After about 1000 iteration steps, the 
norm of the relative velocity error |g” — g"| has fallen to about 7-107° (Figure 
3.25c), which is approximately 4% of the range of each input space dimension, or 
0.08 mm/s for translation and 0.24°/s for rotation. The last 100 of the 2000 itera- 
tion steps are taken to evaluate the velocity error of the trained SOLIM map in 
more detail. Table 3.4 compares the velocity error of the manually trained com- 
bined SOLIM and linear map described above with the velocity error of the learn- 
ing SOLIM map and with the velocity repeatability. The norm of the relative ve- 
locity error of the manually trained map is around 29%, while the norm of the 
relative velocity error of the learning SOLIM map is around 7%, which is a large 
improvement. The repeatability is computed as the mean standard deviation of 10 
velocity measurements with the same voltage amplitudes. The norm of the relative 
repeatability is around 4% and thus in the same order of magnitude as the error of 
the learning SOLIM map. 


Table 3.4. Effector platform: comparison of velocity error of trained SOLIM map, velocity 


error of static map and repeatability. Conditions: /,, = 700Hz, 1000 motion steps per 


step 
measurement. 





x [mm/s] y[mm/s] ¢ [°/s] Ilk 





velocity error 
manually trained map 


0.324 0.344 0.969 


mean [std. dev.] of 100 iterat. [0.201] [0.229] [0.726] 


.. relative to range 16.2%:-2 17.2%-2 16.1%-6 28.6% 





velocity error 
learning SOLIM map 


0.059 0.060 0.299 


mean [std. dev.] of 100 iterat. [0.047] [0.048] [0.227] 


... relative to range 2.97%-2 2.99%-2 4.98%-6 6.53% 





repeatability (10 measurements) 


0.040 0.038 0.151 


mean [std. dev.] of 100 iterat. [0.021] [0.018] [0.062] 


... relative to range 1.99%:-2 1.90%-2 2.51%-6 3.72% 
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Figure 3.25. Effector platform: a. Input space, b. output space, c. error development after 
2000 iteration steps. Conditions: f 


step 


= 700Hz , 1000 motion steps per measurement. 


As already indicated in the introduction of this chapter, the difference in the ve- 
locity error of an actuator controller influences the control performance when posi- 
tioning the effector platform with feedback from a CCD camera. The mean posi- 
tioning error of 100 positioning processes is about 0.0159 mm and 0.0179° when 
using the manually trained map. When using the automatically trained SOLIM 
map, the positioning error is reduced to 0.0041 mm and 0.0101°. Please note that 
all velocity and pose errors are given with respect to the vision system, which pro- 
vides positions with a certain error, but which still allows comparison between the 
two differently trained maps. 
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3.6 Conclusions 
3.6.1 Summary 


The development of this work has been motivated by the problems that appear 
during the development of a controller for mobile microrobots. The main prob- 
lem was the definition of a nonlinear, learning and unambiguous actuation control 
module that yields the necessary actuation parameters for each desired velocity. 
Controllers that are based on self-organizing maps show some interesting proper- 
ties, which qualify them for the application to microrobots, mainly because they 
can learn a smooth controller model and because they are interpretable to a certain 
degree. On the other hand, existing SOM-based approaches still have problems 
when mapping from a desired system output to the corresponding system input and 
when learning an inverse model controller for a given system. 

Details of a closed-loop pose controller for an automated microrobot-based 
nanohandling station are given with the components trajectory controller, motion 
controller and actuator controller. The latter will be replaced by the learning con- 
troller, which is the main contribution of this chapter. 

The SOLIM approach inherits the basic structure of SOM by associating input 
support vectors, output support vectors and topology support vectors, whose com- 
bination is called a node. Important for any SOM-based approach is the definition 
of an order of the nodes, which in the case of SOLIM is done explicitly by assign- 
ing a set of neighboring nodes to each node. SOLIM maps exactly from any input 
support vector to its corresponding output support vector and otherwise inter- or 
extrapolates. SOLIM mapping is based on the similarity of the input vector with 
respect to the input support vector arrangement and the output vector with respect 
to the output support vector arrangement. The SOLIM framework has a separated 
update rule for approximation and for self-organization. The approximation part 
changes the output support vectors such that the cascaded SOLIM map and system 
behavior results in an identity mapping. The approximation update rule is based on 
solving the mapping formula for the value of the output support vector that would 
be most responsible for the measured system output. The self-organization rule 
aligns neighboring output support vectors whose topology vectors are in-line and 
determines the corresponding learning rate depending on their degree of disorder 
and their approximation error. Assuming an ordered arrangement of input support 
vectors, similar input vectors map to similar output vectors. 

The SOLIM approach is validated in various simulations. SOLIM mapping is 
continuous and has a "meaningful" inter- and extrapolation. Its mapping error is 
comparable to other SOM-based interpolation approaches. When the SOLIM map 
is used as a learning controller for a virtual system, e.g., an inverse kinematics 
system, it is able to learn a useful controller, even when its definition is ambiguous 
or when successive desired system output vectors are correlated. 

Finally, the SOLIM map represents an actuator controller, which is trained 
and directly applied to map from desired velocities to amplitudes of the signals 
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driving a microrobot. The training can be manual or automatic and successive 
desired velocities can even be correlated. 


3.6.2 Outlook 


Although some of the problems of current SOM-based controllers, which are iden- 
tified in the introduction, are solved by the SOLIM approach, there is still a gap 
between the current state of SOLIM and the "ideal" inverse model controller. There 
are three main problems of the SOLIM approach that suggest directions for its 
enhancement. Firstly, the current extrapolation method can under certain circum- 
stances lead to distortions. Secondly, SOLIM mapping and learning needs many 
computing operations for a high number of topology space dimensions, and fi- 
nally, the number of nodes for each topology dimension must be known before- 
hand. Besides these extensions, it is also important to use the SOLIM approach for 
other applications and to gain information on possible future research directions in 
the field of SOM-based controllers. 


3.6.2.1 Extrapolation 

The current extrapolation method can be problematic if the input support vectors 
are not arranged on a regular grid, such that the interpolation border and the ex- 
trapolation border are misaligned. While the interpolation border is defined by the 
outer (dC-1)D faces of the d“ D simplicial complex, the extrapolation border is 
defined by the mean differences g7 between each border input support vector and 
its neighbors. There are thus areas outside the interpolation area where the extrapo- 
lation weight is x, =0, and there are areas inside the interpolation area where the 
extrapolation weight is x, > 0. Since the sizes of these distorted areas are related to 
the angles between the extrapolation direction g? of a border node and the normal 
g,,; of a border facet, these distortions are larger when the input space dimensions 
have very different scales. The new extrapolation approach should therefore ensure 
that the extrapolation border is identical to the interpolation border and that the 
mapping result is identical with inter- and extrapolation at the border facets. More- 
over, the extrapolation result must still be continuous, i.e., the transition of the 
influence from one border node or border facet to another must be continuous. 


3.6.2.2 Computational Load 

The number of required computing operations rises faster than exponential with the 
number d° of topology dimensions. There are two main ways of improvement: 
Optimize the calculation under certain constraints or introduce hierarchy and thus 
change the structure of the SOLIM approach. 

The most promising approach for the optimization of the SOLIM computation 
is the reduction of the number of limits that have to be evaluated when mapping 
from an input vector to an output vector. A great reduction of computing time can 
be achieved, if the number d° of input space dimensions is equal to the number 
d° of topology space dimensions, such that an input vector is contained by only 
one d° D simplex. In this case, classic simplex interpolation can be applied, where 
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a search algorithm finds the responsible simplex with the influences of its d° +1 
nodes. 

Another approach for the reduction of the computational load is the introduc- 
tion of hierarchy, which has also been proposed for LLM [25, 46] and SOM [18]. 
In hierarchical SOLIM the responsibility for different input dimensions could be 
divided into separate networks, which are arranged hierarchically, such that each 
node of a network has a subnetwork assigned. Subnetworks at the same level could 
be identical in input space and topology space, such that the input support vectors 
and topology support vectors would be arranged on a regular grid. The output sup- 
port vectors, which are associated to each leaf of the hierarchy, must be different to 
implement a meaningful map. Approximation learning and self-organization learn- 
ing could be performed according to the current SOLIM approach. In this case, the 
nodes with the highest influence on the measured system output g” must be found 
by a hierarchical mapping. First approaches to the implementation of hierarchical 
SOLIM were undertaken in a student project [47] and in a diploma thesis [48]. The 
results show that the number of required computing operations increases linearly 
with topology dimension d“ when using a hierarchical topology with only 1D 
subnetworks, while it increased exponentially when using a d©D network. The 
approximation error that has been achieved in that work is still high compared to 
current non-hierarchical SOLIM. 


3.6.2.3 Predefined Network Size 

Another big limitation of the SOLIM approach is the requirement to define the 
number of nodes in each topology dimension before operation. There are a number 
of methods for growing and shrinking SOM networks, and it would be a great 
enhancement to the SOLIM approach if a method for the dynamic topology con- 
struction could be integrated. Ideally, an inverse model controller should only need 
input and output range and optionally some kind of technical limitation for the 
SOM network, such as maximum computing time per iteration, maximum avail- 
able memory or minimum approximation error. One of these practical criteria 
should then be used to direct the growth and shrinking of the network, e.g., to en- 
sure a uniform approximation error distribution. Special attention should be paid to 
the problem that approximation and self-organizing learning could interfere with 
the growing method. 


3.6.2.4 Applications for SOLIM 

For the further development of the SOLIM approach it is also important to know 
more about the demands of other applications where a learning inverse model con- 
troller can be used. In robotics, and especially in microrobotics, inverse kinematics 
and camera calibration are applications where SOLIM could be applied due to its 
general approach. 

SOLIM could implement inverse kinematics s +> u for different manipulator 
kinematics u > s” , like a voltage-controlled 3-DoF piezo-scanner, which posi- 
tions micro tools. Applying SOLIM for this task is especially useful if the kinemat- 
ics are nonlinear or if the parameter variance of different instances is very large. 
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Applying SOLIM is also practical for the control of different unknown kinematics, 
which are under development. Alternatively, the kinematics could also consist of 
axes with sensors and controllers for each axis, i.e., a device that moves a tool 
centre point to a position s” in world coordinates by moving its axes to local de- 
sired positions s’. SOLIM would then implement the inverse mapping from the 
TCP’s desired world positions sf to the axes’ desired local positions sf. This 
application is practical if the kinematic behavior is position-dependent or nonlin- 
ear. In both cases, SOLIM could adapt to a time-variant kinematic behavior, as- 
suming that data from a global sensor is available for incremental learning. 

SOLIM could also be used for camera calibration by mapping from 3D cam- 
era coordinates s, to 3D world coordinates s,,. This would remove distortions of 
camera mappings s,,+>s,, which are mainly induced by lenses. Moreover, 
SOLIM could even perform this calibration if the camera was moved, assuming 
that this position could be measured. In this case, a hierarchical network would be 
the best solution, where one level covers the camera positions and a second level 
contains subnetworks taking care of the position-dependent camera calibration. 
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4.1 Introduction 


Within the last few years, a trend towards the automation of nanohandling 
processes has emerged. One key problem is the development of a global sensor to 
measure the position of handling tools and nanoobjects during the manipulation 
process. The sensor data is required as feedback to enable the closed-loop 
positioning of the tools and nanoobjects. 

While the manufacturing of nanoobjects like carbon nanotubes (CNT) is mostly 
done automatically using batch processing [1, 2], the production of simple 
nanosystems is teleoperated, due to the incompatibilities of the different batch 
processes. A tool increasingly used to handle micro- and nanoobjects is the atomic 
force microscope (AFM) [3-5]. As mentioned in Chapter 1, one main drawback is 
that the AFM cannot be used simultaneously for manipulation and imaging. Thus, 
no sensor feedback is available, which is required for automation. In [6, 7] an AFM 
has been integrated into a scanning electron microscope (SEM) to image the 
handling process, but it was still carried out using teleoperation. 

Other important tools for the handling of small objects are microrobots with 
piezodrive [8—10], which are equipped with application-specific tools [11, 12]. In 
[13-16], light microscopes are used to gain visual feedback for automated 
microhandling by microrobots. The limit of these systems is the resolving power of 
light microscopes, which does not allow reliable visualization of nanoscale objects. 

The solution proposed in this chapter is to use an SEM, in combination with 
real-time image processing algorithms, as a global pose sensor. The calculated (x, 
y, 9)-pose of handling tools and nanoobjects is used as feedback for the control 
system to enable the automation of nanohandling tasks (Figure 4.1). 
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Figure 4.1. Automation of nanohandling tasks by image processing 


The main benefits of an SEM are high resolution, short image acquisition time, and 
large depth of focus. Especially high frame rates are an advantage, as this enables 
high update rates of the estimated poses for fast closed-loop positioning. However, 
the drawback of fast scanning is strong additive noise. As averaging and filtering 
are too time-consuming, the tracking approaches must provide a high robustness 
against additive noise. 

This chapter is structured as follows: in Section 4.2 relevant properties of the 
SEM are presented and the requirements for the image processing algorithms are 
defined. The software setup and a description of SEM integration in an image 
processing system to enable real-time image access and electron beam control is 
given in Section 4.3. The main part of this chapter is devoted to the implementation 
and validation of real-time tracking algorithms with high robustness against noise. 
In Section 4.4, a cross-correlation approach is described. An active contours 
algorithm with region-based minimization is presented in Section 4.5. 


4.2 The SEM as Sensor 


In addition to Section 2.2, where an SEM is introduced as a sensor for nano- 
handling, the relevant SEM properties for the image processing algorithms are 
described in this section. An SEM is, in terms of resolution (down to 3 nm), image 
acquisition time, and depth of focus, the most suitable sensor for the visualization 
of nanohandling processes. In addition, an SEM offers a large working distance 
and a continuous zoom. The application of scanning electron microscopy to 
different aspects of nanotechnology is described in [17]. The short image acqui- 
sition time is an especially important requirement for the automation of 
nanohandling processes, since high frame rates enable high update rates of the 
estimated poses for fast closed-loop positioning. Because image acquisition is done 
by pixel-wise scanning, the frame rate depends on the image size. The minimum 
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acquisition time depends on the SEM type and varies for current devices between 
125 and 250 ns for one pixel. Thus, a frame rate of 10 fps (Sfps respectively) can 
be achieved with an image size of 1024 x 768 pixels. In Table 4.1 the minimum 
image acquisition times for four different image sizes are shown. 


Table 4.1. Minimum image acquisition times for four different image sizes 

















Image size [pixels] | Acquisition time [ms] Frame rate 
[fps] 
1024 x 768 100 10 
640 x 480 76.8 13 
256 x 256 16.4 6l 
128 x 128 4.1 244 

















As described in the following, the real-time criterion is fulfilled if the pose of 
the target object can be calculated within the minimal image acquisition time. The 
drawback of fast scanning is strong additive noise, as only a few numbers of 
secondary electrons (SE) can be excited during such a short time period. Images 
with low noise are acquired by frame averaging or at lower frame rates (pixel 
averaging). An example of noise reduction by averaging is shown in Figure 4.2. To 
enable automation with high throughput, averaging and frame rate reduction are 
too time-consuming and therefore not applicable. Thus, the image processing 
approaches themselves must be very robust against additive noise. 





Figure 4.2. SEM image of a typical nanohandling scene with an image acquisition time of 
40 seconds (left) and 90 milliseconds (right). The microgripper is described in [12] and [18]. 


Besides additive noise, two further sources of noise are important. The first one 
is gray-level fluctuation, which is a significant issue in SEM. Gray-level 
fluctuations occur due to electrostatic charge and due to variations of the alignment 
of target, electron beam, and secondary electron detector. Especially for pattern- 
matching approaches without invariance against amplitude variations, the 
fluctuations are problematic. 

The third source of noise is clutter caused by objects in the image background 
and/or in the image foreground. Objects in the image foreground can occlude the 
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target object, while background objects often mask edges. As occlusions of the 
target object or the nanotool are not unusual during a handling process, the image 
processing approaches to be developed must be robust against clutter as well. 

Altogether, the following requirements must be fulfilled by the image pro- 
cessing algorithms: 


robustness against additive noise 
robustness against gray-level fluctuations 
robustness against occlusions 

real-time processing 


As the frame rate of an SEM depends on the image size, real-time image 
processing is achieved if the processing time is smaller than or equal to the image 
acquisition time. 


4.3 Integration of the SEM 


A key problem to be solved is the integration of the SEM into the image- 
processing system. To enable real-time tracking it is necessary to access the 
acquired images directly and to control the electron beam. By controlling the 
electron beam a so-called region of interest (ROI) can be used. As the image 
acquisition time depends on the image size, a small ROI optimally adapted to the 
target object can be acquired much faster than a larger one. A direct beam access is 
necessary to control the size and the position of the ROI, which is calculated by the 
image-processing approach. 

Today, most SEMs available on the market are not equipped with a software 
interface for real-time image access and beam control. In most cases, only a slow 
(low bandwidth) remote control interface is available. Therefore, the special image 
acquisition hardware DISS 5 is used. DISS 5 is equipped with a scan generator, an 
image acquisition unit, and a software interface based on a dynamic link library 
(dll). The minimum acquisition time is 250 ns per pixel. The device is connected to 
the external scan interface and to the SE detector of the SEM. Thus, the hard- and 
software to be developed can be used with virtually every SEM that is equipped 
with an external scan interface. The software setup of the image-processing system 
used is depicted in Figure 4.3. As described before, the image data and the beam 
control commands are transmitted to the SEM via the DISS 5 hardware. 

In addition, the LEO 1450 is equipped with a remote control interface that is 
used to change SEM parameters like magnification, working distance, acceleration 
voltage, etc. As the variation of these parameters is not time-critical, the slow 
standard interface can be used. The real-time image processing used to calculate 
the pose of the target objects is the subject of the following sections. 
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Figure 4.3. Software setup of the SEM image-processing system 


4.4 Cross-correlation-based Tracking 


Strictly speaking, correlation-based approaches do not belong to the group of 
object-tracking methods, but belong to the pattern-matching approaches. Tracking 
is achieved by correlating each input frame with an image of the target. 

The cross-correlation of two two-dimensional functions f and g is equal to the 
folding of these functions [19]: 


fog f(-x,-y)* g(x,y). (4.1) 


The correlation operation is denoted by © and the folding operation by *. As a 
folding in the spatial domain is equal to a multiplication in the frequency domain 
(correlation theorem), Equation 4.1 can be rewritten as: 


Jeg =F [Foso]. (4.2) 


G denotes the Fourier transform of g, while F stands for the complex conjugated 
of the Fourier transform of f. In this case, f defines the input image, which contains 
an unknown scene, and g defines a filter mask containing the object to be found. 
The result of the cross-correlation operation is a measure for the similarity of the 
two functions. A high maximum in the result matrix indicates a high similarity. By 
definition of a threshold it can be decided whether the target object is located 
inside the image and at which position. Object recognition by cross-correlation is 
also called pattern matching. In the following, an example clarifies the approach. 
In Figure 4.4 a scene (left) is shown, in which an object (Figure 4.4, middle) is 
searched for. The cross-correlation matrix (Figure 4.4, right) is calculated using 
Equation 4.2. If the maximum value exceeds the defined threshold, the position of 
the maximum indicates the position of the target object in the input image. 


108 Torsten Sievers 


© 
No, = 
Ur 


> 


coefficient 
2 
Ko] 


© 
oo 
wn 





0 150 200 250 


o 50 10 
pixel 


Figure 4.4. The cross-correlation matrix (right) can be used to estimate the (x, y)- 
displacement between an input image (left) and a pattern (middle) 


Usually, the threshold value is determined empirically. The drawback is that, if 
the selected value is too low, false recognitions can occur, whereby a higher value 
will decrease the robustness against noise. Tracking is carried out by sequential 
object recognition in each input frame for an image sequence. 

The advantages of cross-correlation-based tracking are a simple implemen- 
tation and a high robustness against additive noise. In [20], several examples and 
variations of correlation-based approaches adapted to different physical noise are 
described. A disadvantage is the inaccurate modeling of the target object, since 
parts of the background also belong to the model. Thus, this approach is sensitive 
to clutter. A further restriction is that only translations of the target object can be 
estimated. If the target object rotates, deforms, and scales in comparison to the 
pattern (filter mask), the correlation decreases and a direct estimation of the 
transformation parameters is not possible. A quantitative determination of the 
rotation angle and invariance against scaling requires additional patterns that have 
to be folded with the input image. Thus, the computational cost increases signi- 
ficantly. Cross-correlation-based tracking in combination with an SEM as sensor is 
described in [21-22]. In [22], an integrated circuit mounted on a (x, y)-stage is 
positioned and inspected automatically. With the presented approach, the (x, y)- 
position and the orientation can be estimated. The drawback of both techniques is 
slow computation time. In [22] 20 s are needed to estimate the (x, y, g)-pose. A 
further disadvantage is that no adaptive ROI is used. Instead, the SEM is connected 
to a PC via the analog-out interface. 

In the following, an algorithm is described that enables real-time tracking for 
the continuous (x, y, g)-pose estimation by using an adaptive ROI. To decrease the 
time-consumption caused by the orientation estimation with additional patterns, the 
maximum number of correlations per slope cycle is restricted. Therefore, a pattern 
vector with 360 components, each rotated by one degree, is generated. For every 
slope cycle only adjacent patterns (..., pj-1, Pj Pj+1, -..) of the vector are correlated, 
where j denotes the angle. If the starting orientation is known, the pattern p with 
the same orientation and the patterns with +1° (p;-1) and —1° (p;+1) are chosen. The 
pattern with the maximum correlation is estimated and set as p;. After this, the new 
P; and p;+; are selected. The flow diagram of the algorithm is depicted in Figure 
4.5. To detect rotations with more than 1 degree between two frames, the 


Real-time Object Tracking Inside an SEM 109 


orientation estimation is repeated several times for the current frame. If the cross- 
correlation coefficient for pj+ı (respectively p;-ı) is smaller than p,, the correct 
orientation is found. 
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Figure 4.5. Setup for the continuous pose estimation using cross-correlation 


The limit of this approach is that only continuous orientation changes can be 
measured efficiently. In the present application, this prerequisite is fulfilled. 

The cross-correlation approach is validated in consideration of the requirement 
for robustness against additive noise and real-time processing. For both experi- 
ments, the image acquisition time is set to the lowest value, which means that it 
takes 250 ns to acquire one pixel. To measure the robustness against additive noise, 
the standard deviation of x, y, and g is measured for a sequence of 100 frames, 
while the target object is not moved. The result is compared with the theoretical 
accuracy of the approach, which is 1 pixel and 1°. As target objects, a CNT and a 
gripper tip were chosen (Figure 4.6). 

For both objects the accuracy of the pose estimation is 1 pixel for the (x, y)- 
position and 1° for the orientation. Thus, the additive noise does not affect the 
theoretical accuracy, which indicates high robustness for this experiment. 

Real-time processing means that the pose can be calculated within the image 
acquisition time. As the image acquisition time varies with the size of the ROI, the 
processing time is measured for different sizes. The parameters on which the 
processing time primarily depends are the size of the ROI and the number of cross- 
correlations per frame (for the orientation estimation). In addition, the processing 
time depends on the hardware and on the implementation of the FFT. For all 
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measurements, an Intel Pentium 4 standard PC with 2.6 GHz processor was used. 
The algorithm was implemented in C++, and the FFTW3 algorithm was used as 
implementation for the fast Fourier transform [23]. An advantage of the FFTW3 is 
the capability to efficiently process images with sizes not equal to 2”. Therefore, 
the user does not have to use patterns with, e.g., 128 x 128 or 256 x 256 only or to 
expand smaller patterns, which would decrease speed and robustness. 





Figure 4.6. Tracking of a gripper tip and a CNT using cross-correlation and an adaptive 
ROI. The length of the CNT is 6 um 


In Table 4.2, the image acquisition and processing time is shown for three 
different ROI sizes. The processing time is calculated for one cross-correlation step 
and will increase depending on the number of calculations needed for the 
orientation estimation. Real-time processing is achieved if the average number of 
cross-correlations per frame is between three and four. Thus, the maximum 
rotation speed that can be tracked in real-time is approximately 1° per image 
acquisition time, if the highest frame rate is used. 


Table 4.2. Frame rates and correlation calculation time for different ROIs. The correlation 
time is measured for one correlation operation. 
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4.5 Region-based Object Tracking 


The basic idea of this approach is to use an active contour that is minimized to the 
contour of the target object. Active contours have been introduced by Kass, 
Witkin, and Terzopoulos [24] to segment images, e.g., objects from the image 
background. The contour can be implemented by a polygon or spline curve and is 
minimized using an energy function, which considers image features (external 
energy) and the shape of contour (internal energy). Edges, color, gray-level 
statistics, etc. can be used as image features. 

In [25, 26], active contours for visual tracking with edge-based minimization 
are described. The drawback of this approach is a low robustness against additive 
noise, as strong edges are needed for the feature detector. In [27], the application of 
this tracking approach to SEM images is presented. A minimization depending on 
the gray-level statistic leads to higher robustness against noise [28, 29]. This 
approach is also called region-based active contours. In Figure 4.7, two examples 
are shown, where two regions are segmented depending on statistical parameters. 





Figure 4.7. Two examples for segmenting an image in two regions A and B. In the left 
image, the segmentation criterion is the mean gray level. A different standard deviation of 
the gray levels is used as segmentation parameter in the right image. 


In the left image, the active contour segments pixels with higher gray levels from 
the darker background, whereas pixels with equal gray level are segmented in the 
right image. In the following, the energy functions and the minimization algorithm 
needed to use region-based active contours for visual tracking in SEM images are 
described. 


4.5.1 The Energy Functions 


The internal energy function depends on the shape of the contour, whereas the 
external energy function changes with the gray levels of the input image. Two 
assumptions are used for the definition of the external energy function. 


1. The probability of all gray levels in the image is defined by one probability 
function. 

2. The probability parameters (e.g., mean and/or standard deviation) are 
different for the regions to be segmented. 
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This image model is also known as statistical independent region (SIR) model 
[28]. The first assumption ensures that the image noise does not depend on the 
image content, or, in other words, only uncorrelated noise is considered. As this is 
the case for SEM images, the assumption can be made. The second assumption 
ensures that a criterion can be defined for the segmentation. Which probability 
distribution function has to be chosen depends on the sensor and is not significant 
for the SIR model. 

If an SEM is used as sensor, the Bernoulli and the Poisson probability distri- 
bution functions have to be considered (Table 4.3). 


Table 4.3. Probability distributions that describe the gray-level distribution in SEM images 
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In an SEM image, the distribution of the gray levels is described by the Poisson 
distribution [30]. In addition, the Bernoulli distribution is needed to describe binary 
images, which are acquired if only single electrons are detected for each pixel. 

A maximum likelihood estimate j is defined for the parameter set u of the 
given probability distributions that describes a gray level set (image) s: 


f(s) = arg max L(s | u) = arg max | | P(x |u). (4.3) 
H u 


XES 


In Equation 4.3, assumption | is considered and L is the likelihood of s if u is 
given. If an image s is segmented into two regions a and b by an active contour 0, 
the estimated probability Lpseudo(s,8) of s can be calculated by substituting the 
maximum likelihood estimates &, and /,. 


L pseudo (s,0) = L(a | Has 0)L(b | Hy,9) > (4.4) 


with fz pseudo (S,9)ds #1. From this, it is the aim to find the 6(s) that maximizes 


Loredo (s, 0) : 


O(s) = arg max L pseudo (5,0) = arg max (L(a | Âa ,0)L( | Â, +0). (4.5) 


By using Equation 4.3 and Table 4.3, the pseudo-likelihood for a Poisson 
distribution is defined as 
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L pseudo (5:9) = L(a | Mg, 8)L(b | My, 0) 
z I] Poisson (x, Pa JI Ppoisson (x, Ly) i (4.6) 


xea xeb 


After substituting Equation 4.6 in Equation 4.5 it follows that 


A(s) = arg ae L pseudo (s, 0) 


-aema nly Z] mid} 


a xea xeb 


(4.7) 





with f(z) =—zInz. From this, the energy function to minimize is defined as 
1 1 
J pz s,9)= N; — J x +N, — 9x]. 4.8 
Poison ) iz = | N, l | ( ) 


The energy function that considers the Bernoulli probability distribution is defined 
accordingly as 


J Bernoulli (s,0) = waif EDs} mf EDs > (4.9) 


a xea b xeb 





with f(z) =-zInz-(l—z)Ind—-z). 

The energy minimization segments an image into two regions a and b with N, 
and N, pixels. Both energy functions only depend on the sum over the gray levels 
xea,b. An efficient algorithm to minimize the energy function is described in 
Section 4.5.3. 

While the external energy function is mainly used to segment an image, an 
additional internal energy function is needed to smooth the contour. The internal 
energy Jim does not depend on the image but on the shape of the contour. A low 
internal energy leads to a smooth contour, while a complex shape contains a high 
energy. By the weighted summation of both energy functions, the total energy is 
given by 


J(s,9) = Ja (8,8) + aJ;,,,(8) , (4.10) 


where a is a parameter to weigh the internal energy function in relation to the 
external. In the following, four different internal energy functions are described. If 
a polygon is used to implement the active contour, the internal energy function can 
be described as 
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N-1 
Ia@O=> ad. (4.11) 
i=0 


The length of the polygon segment i is denoted by d; The minimization of 
Equation 4.8 leads to a shrinking polygon with equal point distances [29]. 
In [20], an internal energy function is presented, including the curvature of the 


polygon: 
N- 
Jinn (0) = ya (4.12) 
i=0 
with 


d; = p -1P +R). (4.13) 





A further definition of the internal energy can be given by the compactness of 
the polygon [31]: 


R 
J m3 (0) = ae (4.14) 
The length of the polygon is denoted by L and the area by A. This energy function 
supports circular polygons. An advantage of Jj, is the low computational cost, as 
A is calculated with the external energy. 


4.5.2 Fast Implementation 


The external energy is calculated by the summation of the gray levels (Equations 
4.8 and 4.9) located inside and outside the active contour: 


>) sœ y), fork =1,2. (4.15) 


(x,y)eA 


To reduce the computational cost, an ROI A that contains the target object was 
used, as described in Section 4.5.3. Nevertheless, the computing time is still very 
costly due to the number of operations to be carried out for an iterative 
minimization. Thus, an efficient implementation that does not use Equations 4.15 
is required to enable the real-time capability of this approach. 
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The application of the Green—Ostrogradsky theorem 


f| aiv(7 Jaca = [7a (4.16) 


allows a significant improvement of the computing speed. n denotes the vector 
normal to the surface, and div is the divergence operator. The theorem states that 
the surface integral of a scalar function is equivalent to the line integral over the 
contour ò of the surface. Rewritten for the present case, Equation 4.16 becomes 


{| s* (x, yodvdy = | V;.(x, y)é, -Ädt, (4.17) 
(x, y)eA y)ed 


xX, 


with 
Vx) = [sta y)du (4.18) 


for the continuous case and with 


F, (x,y) = dst yy (4.19) 


t=0 


for discrete images [32]. F(x, y) is the sum over the k-th power of the gray levels 
in row y from the left border to column x. Thus, Equation 4.15 can be replaced by a 
summation over the contour 6: 


X sar = Yc. yh 7) (4.20) 


(x,y)eA (x,y)ed 


The coefficient c(x, y) can be —1, 0, or 1 and is defined for each pixel on the 
discrete contour. The lines of the image are divided into sections that belong to A 
and sections that do not belong to A. A section belonging to A is limited by the 
contour 6 on both sides. If the pixels between the limits do not belong to 6, c = —1 
on left and c = 1 on the right boundary. All other pixels of 6 are weighted with 0. 
The sum over c is always 0 for a single row. In Figure 4.8, the coding of the 
contour is clarified with an example. 
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Figure 4.8. Coding of the contour 6 


By using Equation 4.20, the summation of the gray levels inside the contour is 
replaced by a summation along the contour. The precondition is that the contour 
does not contain intersections. A drawback is the additional calculation of F. But F 
has to be calculated only once per acquired image, whereas the energy has to be 
calculated several times per frame as the minimization is done iteratively (see next 
section). Thus, the computational cost is significantly reduced. 


4.5.3 Minimization 


The minimization of Equation 4.10 is carried out iteratively in consideration of the 
transformation space of the target object. In the following, the Euclidean 
similarities are used as transformation space, which contains translations, rotations, 
and scale variations. Thus, the degrees of freedom required for nanohandling are 
covered. 

The minimization is carried out consecutively for all degrees of freedom. First 
the direction (e.g., left or right for the x-position) in which the energy decreases is 
evaluated. Then an interval is estimated wherein the global minimum is located. 
From this, the minimization is carried out stepwise, whereby the interval is 
downsized to increase the accuracy. The minimum interval is three pixels for the 
(x, y)-position. Thus, the maximum accuracy is one pixel. 

Frame-by-frame tracking is achieved by continuous minimization, whereby the 
minimized contour of the current frame is used as the start contour for the 
following frame. In addition to the high robustness against additive noise and due 
to the consideration of the gray level probability distribution, a transformation 
space enables a high robustness against clutter, since only certain shapes of the 
contour are allowed. 

As for the cross-correlation approach, an ROI is used to decrease the image 
acquisition time. The size of the ROI changes with the dimension of the contour 
(Figure 4.9), and the position is shifted with the center of gravity. Thus, the ROI is 
automatically defined for each frame. 
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Figure 4.9. The size of the ROI is defined depending on the size of the contour. 100 pixels 
are added to the maximum dimension 


Because region-based tracking is a frame-by-frame minimization, a first initia- 
lization step is needed to estimate the first ROI. Therefore, the target object has to 
be recognized in the first input image, which can be done by cross-correlation as 
described in Section 4.4. The initial shape of the contour can be defined manually 
or, if available, from a CAD model. In addition, the contour can be initialized by a 
rectangular polygon, if detail shape information is not available. In this case, the 
contour is minimized without a restriction to the transformation space (free 
segmentation) using an approach described in [20]. 

For the (x, y, g)-pose calculation, the parameterization of the contour is used. 
The translation (x, y) is determined by the centroid, which can be defined as [26]: 


L 

f r(s)|r'(s)|ds 
roa , (4.21) 
f r'(s)|ds 


with r(s) = (x(s), y(s))" and s defined on [0...L]. Ir'(s)| is the “throughput speed” 


and 
[ 


defines the length of the contour. For a polygon, Equation 4.21 is rewritten for the 
discrete case 








r'(s)|ds (4.22) 
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ny ZLC) 


N-1 
d; 
i=0 





(4.23) 


y 


(x; yi)” is the center of polygon segment i and d; is its length. The orientation of the 
contour is defined by the principal axes, which are calculated by the moments of 
the contour [31] 


Mog = yD 27S: y). (4.24) 
x oy 
Mp4 denotes the moment of order p+ q . The centered moments are defined as 
Hog = > FX)? OY.) sy), (4.25) 
x y 


with 


mE 2,2,5) _ mg . 
Í Doty OY) Moo 
n 2,2 sey) _ mo. 
PT 


The point (x,, y.)’ denotes the center of the contour and is used as the reference 
point in the following. Centered moments are translation invariant. In addition, 
invariance against scaling is needed, which is obtained by normalizing Equation 
4.25: 





(4.26) 





(4.27) 





__ “pq 
7 pq EZM . (4.28) 
Hog” 
From this, the orientation g of the contour can be calculated by 
2 
p= z arctan — M, (4.29) 
2 120 = 102 
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4.5.4 Evaluation and Results 


In the following, the region-based tracking approach is validated in consideration 
of the requirements defined in Section 4.2. The performance and accuracy 
measurements were performed with a test chessboard as target object, which is 
shown in Figure 4.11. The robustness against clutter and gray level fluctuations is 
validated by a nanohandling task. In every experiment, the image acquisition time 
is set to the minimum value (no averaging, 250 ns per pixel) to test the approach 
with the strongest additive noise. 


4.5.4.1 Performance 

The intention of this section is to demonstrate that the region-based approach 
enables real-time tracking, which means that the pose can be calculated within the 
image acquisition time. As the image acquisition time varies with the size of the 
ROI, the processing time is measured for different sizes. The parameters that the 
processing time primarily depends on are the size of the ROI and the length of the 
contour. A large ROI requires more calculation steps for the estimation of F;, and 
increasing length of the polygon makes the calculation of Equation 4.8 more 
complex. For the performance measurement, it can be considered that the length of 
the contour and the size of the ROI are correlated. Because the shape of the contour 
is restricted by the transformation space, only scale variations can change the 
length. And as described before, the size of the ROI is set depending on the 
contour. Thus, both parameters can be analyzed in parallel. 

The measured image processing time is compared to the image acquisition 
time, because the time changes with the ROI size as well. Within the test sequence, 
the scale of the target object was changed due to a variation of the magnification. 
For all measurements, an Intel Pentium Core 2 Duo standard PC with 2.4 GHz was 
used. The result of the performance test is shown in Figure 4.10. 
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Figure 4.10. Processing in comparison to the image acquisition time. During the sequence 
the scale of the contour and the ROI is changed due to a variation of the SEM magnification. 
At the beginning of the contour the ROI size is 397 x 451 pixels (1), the minimum size is 
243 x 254 pixels (2), and the final size is 766 x 922 pixels (3). 
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Figure 4.10 shows that the processing time is below the image acquisition time 
during the whole sequence. On average, the pose estimation is carried out seven 
times faster than the image acquisition. The difference between processing and 
acquisition increases for larger ROIs. In Table 4.4, the time needed for the 
calculation of F, and for the minimization is shown for four different ROI sizes. 


Table 4.4. Processing time for four different ROI sizes scanned at highest frame rate. The 
processing time is the sum of F; and minimization. 
































Image Processin Minimization 
ROI [pixel] acquisition 8 F, [ms] 
[ms] [ms] [ms] 
243 x 254 15 2 0.2 1.8 
397 x 451 45 9 4 5 
511 x 596 76 11 6 5 
766 x 922 177 13 7 6 








Except for very small ROIs, the minimization and the calculation of F, take the 
same amount of time. In addition, the extra performance enables multiple-object 
tracking even in real-time. 


4.5.4.2 Robustness Against Additive Noise 

The experimental validation of the robustness against additive noise was also 
carried out with the chessboard test sample. Due to the minimization algorithm, the 
position can be calculated with pixel accuracy and the orientation angle with an 
accuracy of one degree. The objective of this experiment is to verify that the pose 
of the target can be calculated with the theoretically possible accuracy even in the 
presence of strong additive noise. To measure the robustness, the standard 
deviation of x, y, and g is measured for a sequence of 100 frames, while the target 
object is not moved. The measurement is repeated for magnifications between 
200x and 50,000x, which is necessary as the noise and blurring increase with 
higher magnifications. To enable a measurement over this large magnification 
range, the target object is changed at 2000x (Figure 4.11). Every measurement is 
done with the shortest image acquisition time. 

The results of the accuracy measurement are shown in Table 4.5. Besides the 
accuracy in pixels, the corresponding sensor resolution in microns is calculated 
using the calibration of the SEM. The accuracy decreases marginally, but persists 
below one pixel up to a magnification of 20,000x. At 50,000, the accuracy of the 
x position is still two pixels or 4.6 nm in world coordinates. For the y position, an 
accuracy of 2.3 nm is achieved. Thus, no significant impact of the additive noise 
on the accuracy can be measured. 
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Figure 4.11. Test object for the accuracy measurement to verify the robustness against 
additive noise. Measurements in the range 200x — 2000x are carried out with a 10 by 10 
chessboard (left image). In the range 5000x — 50,000 a single quadrate with dimension 
1 pm x 1 um is used. 


Table 4.5. Results of the accuracy measurement 





























Magnification | cx [pixel] 6x [um] © [pixel] o, [um] 6, [°] 
200 0.0158 0.5718 0.2095 0.5718 1.198 
500 0.0219 0.2287 0.0186 0.2287 0.3621 
1000 0.063 0.1144 0 0.1144 0 
2000 0.42 0.0572 0.118 0.0572 0 
5000 0.4984 0.0229 0.374 0.0229 1.017 
10,000 0.5086 0.0114 0.5351 0.0114 0.6 
20,000 0.938 0.0057 0.776 0.0057 0.79 
50,000 1.9719 0.0046 1.184 0.0023 0.5276 


























The accuracy of the orientation angle oscillates for different magnifications. In 
general, the accuracy is lower, if the contour contains only a few pixels. However, 
an accuracy of 1° is suitable for most nanohandling tasks. 


4.5.4.3 Robustness Against Clutter 

The validation of the robustness against clutter is done on the basis of a typical 
nanohandling experiment, where a microgripper and a CNT are positioned for a 
gripping task. During the positioning process, the CNT and the gripper mask each 
other. First, the CNT is tracked, while 1/3 is occluded by the gripper. In Figure 
4.12 three images of the sequence are shown. 
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Figure 4.12. Three frames of a sequence wherein a tracked CNT is occluded by a 
microgripper. During the sequence the microgripper is moved over the CNT, while the CNT 
is fixed. The length of the CNT is 8 um. 


In spite of the occlusion by the gripper, the CNT is tracked reliably. In Figure 
4.13, the (x, y)-position of the CNT is shown for 37 frames. 
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Figure 4.13. (x, y)-position of the CNT while it is occluded by the gripper tip 


In a second experiment, one gripper tip is tracked, while being moved over the 
CNT. With this experiment, the robustness against background clutter is tested. To 
increase the difficulty, the magnification is changed while the gripper tip is above 
the CNT. In Figure 4.14, four frames of the test sequence are shown. 
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Figure 4.14. Four frames of a sequence wherein a tracked gripper tip is masked by a CNT. 
In images a and b, the gripper tip is moved over the CNT. The magnification is changed in 
image c, and d. 


As in the experiment before, the target object is tracked reliably even while the 
magnification is changed. 


4.5.4.4 Robustness Against Gray-level Fluctuations 

During nanohandling processes, gray-level fluctuation often occurs when the nano- 
tool touches a surface. The reason is a charge transport between tool and surface. 
In Figure 4.15, an example is shown where a gripper touches a surface. It can be 
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seen that the average gray level increases significantly. To validate the robustness 
against gray-level fluctuations, the position of a CNT is measured while the wafer 
surface is contacted by a gripper. The experiment is repeated three times. 





Figure 4.15. Gray-level fluctuations due to a contact between gripper and surface 


In Figure 4.16 (top), the mean gray-level of the each frame is shown during the 
contacts. Three abrupt jumps can be seen. Directly after the contact, the gripper is 
lifted over the surface and the additional charges drain off slowly. The (x, y)- 
position of the CNT is shown in Figure 4.16 (bottom). In spite of a jump of 
approximately 24 gray levels, the CNT is not lost during the whole sequence. 
While the y-position is not affected by the fluctuation, a displacement of the x- 
position is measured for all three contacts. During the second contact, the y- 
position increases about 11 pixels. The reason is global image drift caused by the 
charge transport, which can be measured by correlation. 


4.6 Conclusions 
4.6.1 Summary 


So far, the handling of nanoobjects is mostly teleoperated, which is costly due to 
time consumption and the required experience. To decrease the costs, automation 
is proposed by combining tools for nanohandling like microrobots with a global 
sensor system. Up to now, the SEM is considered the most suitable imaging sensor 
for nanohandling considering resolution, frame rate, and depth of focus. The main 
drawback of an SEM is the lower image quality due to strong additive noise, if the 
frame rate is high. Thus, the requirements for an image-processing approach are, 
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Figure 4.16. Gray-level fluctuations 


especially, high robustness against additive noise and, in consideration of the 
nanohandling task, robustness against clutter and gray-level fluctuations. In addi- 
tion, invariance against scaling is needed as the SEM is a zooming sensor. 

An important precondition for the proposed sensor system is the integration of 
the SEM into the image-processing system. The proposed solution enables direct 
access to the acquired images and the control of the electron beam, which can be 
performed by a ROI, whereby the tracking efficiency is significantly increased. 

For continuous pose estimation, two image-processing approaches have been 
described. The first one, correlation-based tracking, is very robust against additive 
noise and easy to implement. One drawback is that additional patterns are needed 
to estimate the orientation angle and no invariance against scaling is provided. The 
main drawback is the low robustness against clutter due to the fixed filter mask. 

To overcome these drawbacks and to finally fulfill all requirements, a new 
active-contour-tracking approach has been introduced. The contour is implemented 
as polygon and minimized depending on the gray-level probability distribution. By 
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taking into account the Poisson distribution, a high robustness against additive 
noise has been achieved for SEM images. It has been shown that an efficient 
implementation of the statistical parameter calculation is possible to meet the real- 
time requirement. A high robustness against clutter is achieved by restricting the 
minimization to the transformation space of the target object. The Euclidean 
similarities are an example for a transformation space, which is suitable for most 
nanohandling tasks. 

Subsequently, the region-based tracking approach has been validated in 
consideration of the requirements. Real-time processing has been proven with 
different ROI sizes. The processing time depends on the ROI size and is 7 to 13 
times faster than the image acquisition time. To validate the robustness against 
additive noise, accuracy measurements using the shortest image acquisition time 
(highest noise) have been carried out for different magnifications. A significant 
influence of the additive noise on the theoretically achievable pixel accuracy could 
not be measured. With an SEM magnification of 50,000, the accuracy is up to 2.3 
nm. The robustness against clutter and gray-level fluctuations has been validated 
by a typical nanohandling task, wherein a microgripper and a CNT are positioned. 
Despite a 30% occlusion of the CNT, the measured pose is not influenced. Gray- 
level fluctuation has been induced by a contact between microgripper and wafer. It 
has been shown that only the global image drift is measured. 

All requirements defined in Section 4.2 have been fulfilled by the region-based 
tracking approach. In combination with the SEM as image sensor, a high- 
resolution sensor for several applications in the field of nanohandling is available. 


4.6.2 Outlook 


The results described in this chapter are very promising and will trigger more 
research work in this direction. Beside the continuous pose estimation of nanotools 
and -objects, additional information, estimated by image-processing algorithms, 
could increase the efficiency of automated nanohandling processes. In addition to 
the automation of the handling tasks, it is important to automate the quality control 
as well. Therefore, an automated analysis of the contour parameter is proposed. 
The calculated data can be used to verify the alignment of a target object in the 
gripper or at the target position after the place operation. Future work will also deal 
with the integration of other existing SEM detectors. In addition to the SE detector, 
a backscattered electron detector (BSE detector) and an energy dispersive X-ray 
detector (EDX detector) can be used to analyze the material distribution of 
samples. For example, an automated analysis of the detector signals by image 
processing can be used for an automatic quality control of bonding processes. 
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5.1 Introduction 


Handling processes with the aim of changing the position of objects relative to 
each other can be carried out by a microrobot-based nanohandling station (Chap- 
ters 1 and 2), which makes the teleoperated or automated handling of nanoobjects 
possible. Because the object size is in the range of um, sub-um, and even down to a 
few nm, scanning electron microscopes (SEM) are increasingly employed to 
observe these processes. They allow enormous magnifications to be achieved, so 
that manipulation processes on the nanometer scale can be observed. 

Through observation, information such as the position of the objects and of the 
tools can be determined. This information is needed by the user or by a robot 
control system. It is therefore important to know the relative position of the objects 
and tools in all three dimensions of space. In addition, the determination has to be 
carried out precisely, robustly, fast, and without interfering with the handling 
process. 

Commercial standard SEMs deliver 2D images without depth information. 
However, for precise handling and working without disturbing the object, 3D 
information is needed. This chapter considers various possibilities to obtain 3D 
information from SEMs with respect to nanohandling actions. Section 5.2 de- 
scribes some of the basic concepts from mathematics, biology, and microscopy, 
which are necessary for the understanding of the following chapter. In Section 5.3, 
current systems used to obtain 3D information using SEMs are presented. Finally, 
a 3D imaging system for nanohandling is described in detail in Section 5.4, and 
Section 5.5 presents its use in the manipulation of nanoobjects, e.g., nanotubes. 
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5.2 Basic Concepts 


In this chapter, the most important basics of this topic will be described. In Section 
5.2.1, an overview of the principles of stereoscopic image approaches is provided. 
In Section 5.2.2, the principles of stereoscopic imaging for SEMs are shown and in 
5.2.3, the mathematical basics are presented. Section 5.2.4 gives an introduction to 
the subject of biological vision systems. 


5.2.1 General Stereoscopic Image Approach 


The stereoscopic principle is described here with the help of the human visual 
system. This enables depth perception by using two eyes, which observe the 
surroundings from slightly different perspectives. This shift allows the brain to 
reconstruct the depth data. 

Figure 5.1 shows a schematic representation of the principle function. The eyes 
focus on a fixed point (point F), which thus impacts on the middle of the retinas of 
both eyes. The optical axes intersect at point F. If the retinal projections of point P 
are considered, which is at the same distance from the observer, it can be seen that 
relative to F, point P appears to be equally shifted in both the left and the right eye. 
On the other hand, point Q is placed in front of F and therefore results in dissimilar 
shifts in the left and right eyes. In this example, the magnitudes of the shifts are 
equal, but not the directions. The difference in the shifts between the left and right 
eyes is known as the disparity. 

For P, the disparity (F; - Pi) — (F, - P,) = 0. Points with a disparity of 0 have the 
same distance from the observer as the fixed point. 

For Q: (F; - Qi) — (F, - Q,) # 0. Depending on whether the disparity is positive 
or negative, the point is located either in front of or behind the fixed point. 


F P 
vergence 4 
angle ———~ á 






P XE 
'O, F, P, FÊ. 
left eye right eye 


Figure 5.1. Schematic diagram of the stereoscopic effect: F — fixed point; P — a point with 
equal disparity to F; Q — a point with a different disparity 
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The disparity is a relative measure of the distance. With this alone, it is not 
possible to determine the absolute distance, because the absolute distance is 
dependent on other parameters such as the distance of the eyes from each other and 
the angle of vergence (see below). For the calculation of absolute object coor- 
dinates from measured disparities and for a detailed consideration of geometrical 
relationships, information can be found in [1]. In the context of this chapter, only 
the determination of the relative distance or depth is considered. 

The problem of the (relative) determination of depth is therefore reduced to the 
determination of the disparities of all points in an image. The objective then is to 
find the corresponding disparity values for as many points as possible, in order to 
include the relative depth relationships in the resulting image. 


5.2.1.1 The Cyclopean View 
Although humans possess two eyes, they perceive only one image of the 
surroundings. The brain combines the two images with their slightly differing 
perspectives to form a complete image (fusion). The resulting image has a 
perspective corresponding to a single eye positioned between left and right. This is 
known as the cyclopean view. 

Considering disparities, the question arises, which view should these relate to? 
If a point in the left image is observed, it appears shifted to the left in the right 
image (positive disparity). On the other hand, observed from the right image, it is 
shifted to the right in the left image (negative disparity). It is therefore important to 
differentiate between left, right, and cyclopean view and the corresponding 
disparity maps. Some algorithms, e.g., in [2], use bidirectional matching, in which 
two disparity maps can be generated and where the results relate to the left or right 
view. Here, one of the input images is used as the main image and the other as 
reference image. In [3], on the other hand, the input images undergo a cyclopean 
analysis, that is, the determined disparities are based on the cyclopean view. This 
makes the construction of the image with a cyclopean view possible. 

It is assumed in this chapter that the disparities are based on the cyclopean 
view. The calculation of the disparities at an image location therefore describes a 
pixel coordinate from a cyclopean view. 


5.2.1.2 Disparity Space 

Disparity is a local attribute of an image, meaning that at different parts of a 
stereoscopic image pair, different disparities can arise. These can be both positive 
and negative. If we take dmin and dmax to represent the minimum and maximum 
disparities in a stereoscopic image pair, then the value range of the disparities [dynini 
dmax] is called the disparity space. 

In the calculation of the disparities using algorithms, all procedures have in 
common that they can only look for a limited number of disparities. In most cases, 
the search area is limited, so that the calculation time is kept to a reasonable level. 
The area of values for the disparities, which is searched for, is known as the 
‘disparity search area’. If a disparity arising from an image lies outside of this 
search area, it cannot be detected. The disparity search area should therefore 
completely cover the disparity space. 
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According to the research literature, negative disparities relate to points which 
are located in front of the fixed point and positive disparities to points behind the 
fixed point [4-6]. Without prejudice to the general approach, this chapter takes the 
opposite position, as the points needed for the disparity map should be brighter the 
closer they are to the observer (brighter points are represented with higher pixel 
values and darker ones with lower values). The disparity region of a stereoscopic 
image pair can be inverted, so that left and right sides of the image exchange 
positions. 


5.2.1.3 Vergence and Version 

The angle at which the optical axes of a visual system intersect is called the 
vergence angle. It is dependent on the distance from the fixation point. The closer 
this is, the bigger is the angle of vergence. For example, if the eyes in Figure 5.2 
focus on point Q, the vergence angle is larger than for a focusing on point F. 
Depending on the vergence angle, the eyes are focused more in either an inward or 
outward direction. Although the eyes move with a change in the vergence angle 
(that is, the fixation distance), this has no effect on the cyclopean line of sight, 
known as version. Similarly, the line of sight has no effect on vergence. The 
following diagram illustrates this. 





Figure 5.2. The Vieth—Miiller circle 


Figure 5.2 shows the Vieth—Miiller circle arranged for the left eye, right eye, 
and fixation point. In the figure, a represents the vergence angle and y the version 
angle, which corresponds to the cyclopean line of sight. The angles of the optical 
axes for each eye relative to the direction of the head are indicated by g; and g,. 
According to Thales’ theorem, a is equal for all points F on the Vieth—Miiller 
circle. A change in version has therefore no effect on vergence. This can also be 
seen by considering ø; and ø, (that is, the position of the axes). The angles are 
related as follows: 


Q =P- P,» (3.1) 
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y= 50 +9,). (5.2) 


If g; and g, change in harmony (that is, the eyes turn equally to the left or right), 
then only version y, the line of sight, changes; the vergence a remains constant. If 
gı and ø, change inversely (the eyes turn inwards or outwards — convergence or 
divergence) only the vergence a, the fixation distance, changes. The version y 
remains constant. If only one eye turns, both version and vergence change. 

To enable the reconstruction of depth data, vergence is of particular importance 
as it affects the fixation distance. The cyclopean image cannot be created with a 
simple overlaying of two partial images. If both partial images are being overlaid, 
only limited areas of the images merge (local merging). 

The control of the eye muscles is closely linked to disparity detection. If the 
eyes have to focus on a new fixation point, the required eye movement can be 
calculated (or at least estimated fairly accurately) based on the actual measured 
disparities. For example, if a point is fixed, which lies further away (but in the 
same cyclopean line of sight), then the vergence signals, which control the 
movement of the eyes, are determined by the measured disparities of the new 
fixation point relative to the actual fixation point. Figure 5.3 shows an example of 
this. In Figure 5.3a, the eyes are focused on the level F,. Points on this level have 
disparity 0. 


d(F,) 0 F, 
0 F, d(F) 
d 
d 
a b 


Figure 5.3. Focusing on two different depths: a. eyes are focused on level F}; b. eyes are 
focused on level Fy 


The points on the more distant level F, have a negative disparity d(F2). The 
greater the extent of this disparity, the further do the eyes have to turn outwards in 
order to focus on the level F, (Figure 5.3b). Through the changes in the fixation 
level, the disparities in the new level become 0. The disparity region is shifted; it is 
always relative to the fixation point. In Figure 5.3a, F always has a negative 
disparity to F;, In Figure 5.3b, F, always has a positive disparity to F>, 
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5.2.1.4 Vergence System 

As referred to above, the disparity search area for the algorithmical processing of 
stereoscopic images has a limited extent. Disparities outside of this search area 
cannot be detected. Because the value range of the arising disparities in a 
stereoscopic image pair is not known in advance, the question arises as to how the 
search area can be selected. One possible answer is to make the search area 
sufficiently large such that the probability of disparities lying outside the area is 
reduced. However, as the calculation time is strongly influenced by the size of the 
search area, this is not a good solution. 

The use of a vergence system is therefore sensible, i.e., a system that deals 
with the orientation of an image pair to each other. The idea here is to analyze a 
stereo image pair with a coarse resolution, which reduces the disparities arising in 
the image data so that the area of detectable disparities is effectively extended. 
Naturally, the disparity calculation for coarse resolutions is less precise, but it 
serves as the rough orientation of the images, so that a finer resolution can be fixed 
for the disparity search area. Figure 5.4 shows this schematically. 
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Figure 5.4. Coarse selection of a fixation point prior to a more exact calculation: a. larger 
depth range can be included; b. selected disparity search area; c. selection of a additional 
fixation point 


Through the analysis of the images with a coarse resolution, the effective 
disparity search area is increased, that is, a larger depth range can be included 
(Figure 5.4a). After this, the disparity search area can be selected for fine 
resolutions so that the disparities arising in the image data are included in the 
search area (Figure 5.4b). If there are still disparities outside the search area, these 
can be relocated through the selection of a further fixation point so that the missing 
disparities can be recognized (Figure 5.4c). 
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5.2.2 Principle of Stereoscopic Image Approaches in the SEM 


5.2.2.1 Structure of the SEM 
Because of the “long“ wavelength of light, it is not suitable for the observation of 
structures on a nanometer scale (Chapter 2). In this subsection, a short overview of 
the structure of the SEM is given. This is needed for stereoscopic image gene- 
ration, which is clearly described below. More details can be found in Chapter 10. 
In the SEM, the sample to be examined is scanned with electrons instead of 
photons, and the resulting reaction is measured. The whole of the sample is not 
scanned with electrons at once, but individual points are scanned one after the 
other in a grid pattern before (in modern equipment) being composed into a digital 
image. Figure 5.5 shows a schematic representation of a scanning electron 
microscope. The complete setup is under vacuum during operation, mainly to 
allow the electrons to move freely and not be obstructed or diverted by gas 
molecules [7]. 
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Figure 5.5. Schematic representation of a scanning electron microscope 


The operating mode of an SEM can be described as follows: 


e Creation of an electron beam. An electron cannon is the source of the 
beam. The free electrons generated at the cathode (such as a tungsten 
filament) are accelerated through an electric field and shot at the sample at 
high speed. 


136 Marco Jaéhnisch 


e Electron optics. In order to control the beam, different lens systems are 
used. These normally consist of charged coils to generate the magnetic field 
through which the electrons can be deflected. (Note — the actual lenses are 
not solid, that is, they consist of magnetic fields. The lenses shown in Figure 
5.5 indicate only that the function is similar to that of an optical lens). The 
condenser lenses and apertures have the function of bundling the beam, and 
the quality is frequently improved by the use of several lenses. Finally, the 
beam is deflected through the scanning coils and focused via the objective 
onto a sharply defined point on the sample. In this way, different parts of 
the sample are scanned, one after the other. 

e Interaction with the probe. When the beam of electrons strikes the 
sample, several types of interaction of result. The electrons which are set 
free allow inferences to be drawn about material qualities. The beamed 
electrons are backscattered, as are so-called secondary electrons which are 
knocked out of the sample. These give information about the topography of 
the material. 

e Detector. Different types of detector are used depending on the information 
required to be collected. The most frequent type used (also in the context of 
this work) for topographical data is the secondary electron detector. The 
number of electrons knocked out at each contact point determines the 
brightness of the corresponding point on the image, which is also why only 
gray-toned images are produced. 


SEMs are used mainly when the resolution capability of a light microscope is not 
adequate. Light microscopes achieve magnifications up to 1500 times. On the other 
hand, SEMs achieve magnifications of up to 400,000 times and more. As a result 
of the bundling of the electron beam and the use of very fine apertures, images 
arise that have a very high depth of focus compared to light microscopes. Because 
the sample is sequentially scanned, there is a time delay in the generation of the 
complete image. Thus, there is always a compromise between image quality and 
the speed of image creation. Images are generated in less than a second (low scan 
speed), for example, are usually particularly noisy. Images of very high quality 
(high scan speed) may require 30 seconds or more. 


5.2.2.2 Generation of Stereoscopic Images in the SEM 

For the generation of an image, the SEM scans individual points of the sample 
sequentially. This makes the generation of a simultaneously synchronized stereo 
image pair impossible. Even if it were technically feasible to apply two electron 
beams alternately activated during the scanning process, this would be out of 
proportion to the technical and financial expenditure required. Therefore, the 
images of a stereoscopic image pair must be sequentially generated in a process in 
which the angle of observation of the sample is changed between both scans. This 
can be done in two basic ways: by tilting the sample table or by deflecting the 
electron beam (Figure 5.6). 
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Figure 5.6. Generating a stereoscopic image in an SEM 


Sample table tilting: it is possible to generate an image pair by tilting the 
sample table along with the sample between the scans. The effect of this is 
that the electron beam hits the sample from two different angles, resulting in 
two different perspectives of the sample. The problem with this method is 
with respect to the concentric tilting. It cannot generally be assumed that the 
actual tilting axis of the sample table is at the same height as the sample 
itself. In the case of the SEM used for this work (LEO 1450), the actual 
tilting axis is clearly under the carrying medium. This results in the sample 
disappearing from the area of impact of the electron beam during tilting 
(Figure 5.7). Because of this, for concentric tilting to work, the sample table 
has to be translatively corrected around the physical axis after tilting [8]. 
Tilting the sample table results in the sample moving to or away from the 
detector, so that more or fewer electrons, respectively, are measured. 
Images produced in this way therefore normally have a contrast difference 
[9]. 

Beam deflection: another method of generating an image pair is to deflect 
the electron beam. This means that the position of the sample is not 
changed, but instead, the electron beam is deflected by an extended lens 
system causing it to hit the sample at a different angle. The differences in 
angle allow for two different perspectives of the sample. The advantage of 
this method is that it is not necessary to make mechanical changes and so 
the angle of view can be very quickly changed. Of course, the lens system 
must be very precise to ensure that the electron beam hits the same place on 
the sample in each case. In [10-12], a specially developed electron column 
with integrated lens system, which allows for tilting at several angles, is 
used. 
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5.2.2.3 Influences on the Disparity Space 

The greater the disparity space in a stereoscopic image pair is, the greater the 
disparity search area that has to be selected so that all disparities can be found. 
Because the size of the disparity search area has a direct influence on the 
processing time for an image pair, the disparity region should be kept to a 
reasonable size. This is dependent on three factors. It is limited by the highest and 
lowest points of the sample the tilt angle, and the set magnification. This is 
exemplified in Figure 5.7a, showing a side view of the sample table on which a 


sample is placed. 
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Figure 5.7. How the disparities are affected by the height of the point, the tilt angle, and the 
set magnification: a. side view of the sample table; b. tilting the sample; c. altering the 
points on a scanline by changing the magnification 


The highest point of the sample P and the lowest point Q are observed. Tilting 
the sample means that both points (as shown in Figure 5.7b), because of their 
different heights, move different distances, and this is observable in stereoscopic 
images as disparities. In this example, P will have a positive disparity d,, because it 
lies above the tilt axis K, and Q will have a corresponding negative disparity d}. As 
P and Q represent the highest and lowest points of the sample, the disparity region 
is limited to d, and d,. If the tilt angle is increased, the disparity region increases 
as well (and decreases in the case of a reduction). The same happens in the case of 
the magnification factor. Figure 5.7c shows schematically the places on a scanning 
line where a point shifted by tilting is imaged, when the magnification is changed. 
Therefore the change of magnification also affects the disparity region. 
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5.2.3 Mathematical Basics 


In this section, the most important mathematical basics are described. The 
relationship between these basics and the stereoscopic image processing will be 
explained later in the work. 


5.2.3.1 Convolution 

A convolution is a mathematical integral defined by two functions f(x) and g(x) 
which produce a result function h(x). In a continuous case, h(x) is the integral over 
the product of f(x) and a mirror-image shifted version of g(x). With discrete 
functions, the integral is a sum, of 


A(x) =(f ® g(x) = Df): gen), (5.3) 


Convolutions are frequently used as a filter operation in image processing. The 
image is presented as a two-dimensional function f(x,y) and filtered through a 
convolution mask g(x,y). Different convolution masks are used for different 
filtrations. For example, the convolution mask: 


Lol RolRwole 
OlRolRwole 
ol RolRwolre 


has a smoothing effect, which suppresses image noise but also makes the image 
less sharp. Such masks are known as smoothing masks. 

Calculations involving convolutions are normally time-consuming and require 
a large number of steps, because for every value of h(x), it is necessary to calculate 
a sum of products, even though function values of a convolution mask frequently 
have value 0, meaning that it is not necessary to carry out all the operations. 
However, the two-dimensional functions arising in the image processing increase 
the calculation effort considerably. 


5.2.3.2 Frequency Analysis 

According to the French mathematician Jean B.J. Fourier, any periodical signal in 
a continuous sequence of harmonic (sinusoidal) signals is composed of different 
frequencies, amplitudes, and phases. In order to describe the signal over time, 
either the value of the amplitude at that point in time can be given, or the 
amplitudes of frequencies occurring in the signal can be given with the 
corresponding phases. 

For the usual system of representing images in the so-called spatial domain, 
each image coordinate is assigned a brightness intensity. In this way, each image 
can be clearly described. In image processing, images are also frequently 
composed as (discrete) two-dimensional spatially moving signals. The image data 


140 Marco Jaéhnisch 


can then be represented through a Fourier transform as a so-called frequency 
domain, in which an image is described only in terms of the occurring frequencies 
and their associated phases. 

For the data content of an image, it is the same whether this is given in the 
spatial or frequency domain. The data does not change; it is simply presented in a 
different way. However, different ways of presenting the data may be more or less 
suitable to show particular contents or to allow for more efficient processing. 


unfilten: filtered 






spatial domain 


frequency domain 


multiplication 





c d 
Figure 5.8. An example of image smoothing: a. initial image in spatial domain; b. a filtered 


image in spatial domain; c. frequency domain; d. a filtered image in frequency domain; e. 
convolution mask in spatial and frequency domain 


Figure 5.8 shows an example of a smoothing operation to illustrate the point. 
Normally, the initial image (a) would be convoluted with, e.g., a Gaussian 
smoothing mask (e) in order to achieve the desired results (b). A smoothing 
operation needs, however, a lot of effort, so it is helpful to avoid it. Therefore, the 
image is first transformed into the frequency domain, as shown in (c). (The 
frequency images (c) and (d) are brightened here for simplification.) In this 
representation, the brightness of a point represents the amplitude of an occurring 
frequency in the initial image. The lower frequencies are located in the middle 
region, and the higher frequencies in the outer region. Because of the high level of 
noise in the initial image, a lot of high frequencies are present. Smoothing the 
image corresponds to a low-pass filter in which high frequencies are suppressed. 
This is simple to achieve in the frequency domain as the frequency spectrum can, 
for example, be multiplied by a Gaussian function (d). Through a subsequent 
inverse transformation into the spatial domain, the same result arises as for the 
convolution of the initial image in the spatial domain. To achieve an equivalence of 
both operations, the convolution mask must be transformed likewise. The 
particular point about the Gaussian smoothing mask used here is that in the 
frequency domain, a Gaussian function is also represented, as shown in (e). 
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The conversion of the image data from spatial to frequency domain is done 
using the Fourier transform, in which a corresponding frequency and phase 
spectrum results. The following representation relates to the discrete two- 
dimensional Fourier transform, as used in image processing: 


1 M-1 N-I x vy 
KE faye MN. (5.4) 


f(x,y) represents the image in the spatial domain, F(w,v) in the frequency domain. 
Here, u and v are frequencies occurring along the x or y axes. M and N are the 
dimensions of the image in pixels. An important feature of this function is that it is 
reversable. The inverse Fourier transform is calculated by: 
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It should be noted that F(u,v) is a complex value and contains frequency as well as 
phase data. For the example shown in Figure 5.8, the phase data is not interesting, 
which is why only the frequency spectrum of the transformed image is shown. The 
phase data can, however, be used in the context of disparity detection. 


5.2.3.3 Gabor Function 

Gabor functions, named after the physician D. Gabor, are often used in the 
frequency analysis of image data. Basically, a Gabor function is a (co-)sine wave 
under a Gaussian bell. In a one-dimensional case, the Gabor function is described 
by 


2(X30,9,0) = ew no cos(@x + Q), (5.6) 
where œ is the frequency and ø is the phase of the cosine function. ø indicates the 
width of the Gaussian function. Figure 5.9a gives an example of a one- 
dimensional Gabor function. 

In the two-dimensional case, the additional parameter @ describes the 
orientation of the function: 


'2 '2 





X +y 
2(x,V,0,0,90,0)=e 2° -cos(@x +9), 
x =x-cos(@)+ y-sin(9), (5.7) 


y =—x-sin(9)+ y-cos(@). 


Figure 5.9b and c show a two-dimensional Gabor function in two different 
orientations. 
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Figure 5.9. Examples of Gabor functions: a. one-dimensional; b. and c. two-dimensional in 
different orientations 


The main advantage of these functions is that they are limited in both the 
spatial domain and the frequency domain (Figure 5.10a and b). They are therefore 
suitable for carrying out a spatial band-pass filtering of image data. In this 
process, 6 influences the bandwidth and œ the middle frequencies of the filter. If 
the frequency spectrum of an image was multiplied by, for example, the function 
shown in Figure 5.10b, certain frequencies with a particular orientation at a local 
region of the image would be filtered out. 

For digital image processing, these functions are made discrete, i.e., scanned at 
discrete locations. In this way, there is a finite number of filter coefficients which 
can be used as convolution masks for filtering image data. 





a b 


Figure 5.10. A two-dimensional Gabor function in a. spatial domain and b. frequency 


domain with image size 256 x 256 pixels; à = cia 16 pixels, 0=0°,9=0°,0 x 9 
© 
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5.2.4 Biological Vision Systems 


For many problems known to science, there are analogies in nature. Humans have 
increasingly made use of natural systems and processes which have been perfected 
by nature over millions of years. To determine, how disparity information can be 
subtracted from stereoscopic image pairs, it is apposite to examine biological 
vision systems. The following section deals in short with the functioning of the 
neurons which make up biological vision systems. 


5.2.4.1 Neuron Models 

The neurons in a neuronal network are connected to each other in a complicated 
way. If they receive sufficient stimulation, they are able to fire off so-called action 
potentials (APs) which are transmitted to the next neuron, where they may cause 
another fire-off. A neuron normally receives APs from several other (precursor) 
neurons and then transmits its APs to several other (successor) neurons. A neuron 
is schematically represented in Figure 5.11. The number of precursor neurons from 
which the neuron receives APs, and which therefore influence its response, is 
known as the “receptive field”. 


AP 


Figure 5.11. A schematic representation of a neuron 


The APs arriving from the neurons of the receptive field have varying impacts 
on the behavior of the neuron; they are weighted. This weighting can also be 
negative; that is, having an inhibiting effect on the neuron. Whether a neuron fires 
off an AP to its successors or not depends on the weighted sum of the incoming 
potential. 

In fact, the exact firing behavior of a neuron is not just dependent on the 
weightings, but in addition on the timing at which the APs arrive. The behavior of 
a group of neurons is therefore particularly complex. It is still not completely 
understood how neurons code and process information. 

The neuron model which is used here is a simplified version. The firing rate, 
that is, the (mean) number of action potentials which are set off per unit of time, is 
the measure for the firing of a neuron. The firing rate f of a neuron and the 
incoming firing rates of the precursors fi, ..., fa with the weights gi, ..., Zn are 
calculated by: 


f= S (5.8) 
i=l 
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Because f can have a negative value but neurons cannot code a “negative” firing, 
this is a simplification of the situation in which a value is described by two 
neurons, one for positive values and one for negative values, whereby only one of 
the two is able to fire [4]. 


5.2.4.2 Depth Perception in Biological Vision Systems 

During the earlier period of research into stereoscopy, it was assumed that the 
impression of depth that humans received from the surroundings based on 
disparities had mainly a geometrical cause and arose from the different spatial 
projections of lines or object edges. In [13], experiments were carried out in the 
1960s with so-called random dot stereograms and it was shown that even for 
structureless images, a perception of depth arose. Observed individually, they 
consist of randomly arranged points with no order and indicating no structure. Seen 
stereoscopically, however, depth structures can be recognized. 

The fact that the human visual system is able to abstract disparity information 
from such images leads to the conclusion that this is not only a result of the 
analysis of shapes or colors. In addition, the experiments showed that the 
recognition of disparities takes place relatively early in the visual data path of 
biological perception systems and not only after a time-consuming monocular 
analysis. 

In [14] experiments with cats showed that there are neurons in the visual cortex 
which are sensitive to light stimulation of particular types. In the main, a 
differentiation is made between two groups of neurons: simple cells and complex 
cells. The reactions of individual neurons have been measured experimentally in 
[4]. They suggested a model which would describe the behavior of these neurons 
and which can be used for the investigation of disparities. It was found from this 
that the receptive fields of simple cells could be modeled using Gabor functions. 
This leads to the suspicion that the brain carries out a frequency analysis to 
determine disparities. “[...] the brain has at its disposal the two-dimensional 
Fourier transform of the presented brightness distribution” [15]. 

The model developed in [4] is described as an ”energy model”. Because it has a 
central importance in this work, it will be described more fully in the following 
section. 


5.2.4.3 Energy Models 

For the calculation of signal energies, energy models use so-called energy 
neurons. Figure 5.12 shows a schematic outline of such a neuron. It is made from 
two simple cells and one complex cell. The receptive field of a simple cell 
stretches over a limited region of the retina. The input weights are described 
through Gabor functions, that is, the input stimulus (a limited local area of the 
retina) is weighted through Gabor coefficients. The resulting sum represents the 
response of a simple cell. Such a neuron is described as a linear neuron. 

An individual linear neuron cannot be used as a measure for the signal energy 
as its response is dependent upon which phase of the stimulus from the receptive 
field arrives. The example of a sinusoidal stimulus is considered, which has the 
same frequency and phase as the Gabor function which is used. In this case, the 
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response of the linear neuron is maximized. If the phase of the stimulus shifts, the 
response is reduced. 

Through the use of two simple cells, the construction of a response which is 
independent of the stimulus phase can be construed. Both simple cells possess the 
same receptive field but filter the stimulus with different Gabor functions, the 
phases of the functions differing by 90°. Such a pair of filter functions is described 
as a quadratic pair. The response of the two linear neurons is squared by a 
complex cell and summed in order to maintain a constant measure of signal energy. 
This construction is helped by the fact that sin? (x) + cos? (x) = 1 is valid. 

Energy neurons can be used for the detection of disparities, since they can be 
extended to binocular energy neurons (Figure 5.12b). 








a b 


Figure 5.12. a. Structure of an energy neuron; b. structure of a binocular energy neuron 


Here, a simple cell receives signals from both the left and right image, which 
are filtered with separate filter functions. If we observe the phase at which a 
stimulus arrives at the retinas in both the left and right eye, we see that the 
response of a simple cell is maximized when the phases are the same [16]. This 
corresponds to a disparity of 0. If there are differences in the phases, the disparity 
is not equal to 0 and the response is reduced. Because the response of a simple cell 
is not just dependent on the relative phase difference, but also additionally on the 
absolute phase as in a monocular case, a quadratic pair is also used here. A 
complex cell squares and sums the response of two simple cells which have a filter 
function differing in the phase by 90°. 


5.3 Systems for Depth Detection in the SEM 


Various methods are used to determine depth of objects in micro- and nano- 
handling operations in SEMs. In addition, in the field of materials research, 
surfaces with different properties are characterized. The following section presents 
an overview of the available procedures described in the literature. 
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5.3.1 Non-stereoscopic Image Approaches 


In [17], the use of a so-called touchdown sensor is described. It consists of two 
flexible piezos, whereby the first serves as an actuator and the second as a sensor. 
A current is applied across the actuator which causes this to oscillate at its own 
frequency. The oscillations and their amplitude can be measured by the second 
piezo through a small phase shift. At the tip of the sensor, a nanogripper is 
attached. If the gripper now comes into contact with another object, its shape 
deforms. This leads to a sudden change in the measured amplitude at the sensor. 
This change can be measured and thus serves as a contact detector. One difficulty 
in using this procedure is that there is contact between the gripper tip and the 
object to be gripped. This can lead to damage of the tool or object because of the 
oscillations. 

In [18], another approach is described. Here, the SEM is equipped with an 
additional light microscope for automatic object manipulation. They use a 
simplified experimental design in which spherical objects with a diameter of 30 um 
can be positioned according to a set pattern. The objects are lifted and placed using 
a needle-shaped micromanipulator. The electron microscope is used to deliver the 
top view of the objects. By means of an edge detection algorithm, the 2D position 
of the objects is determined. The additional light microscope, which is mounted in 
the sidewall of the SEM, delivers a side view of the objects. Therefore, it is 
possible to detect whether the object is lifted up or not. Due to limited resolution of 
the light microscope and the fixed magnification of 1000x, only relatively large 
objects can be observed. 

In [19] a triangulation approach for the determination of the third dimension is 
used and two methods are proposed. In the first method, a light microscope is used 
to project a line from a laser onto the object surface at an angle. The relative depth 
can be calculated through the relative shift between the base line and the projection 
line. The higher the shift, the higher is the object. The surface of the object must 
first be treated with a luminescent material. When the beam hits the object surface, 
a luminous spot appears, which is detected by a miniaturized light microscope that 
is mounted inside the vacuum chamber. Due to the resolution limits of the light 
microscope (CCD camera), the accuracy of this approach is 1-5 um. 

In [20], a four-quadrant backscattered electron (BSE) detector system for 
the three-dimensional surface characterization of microstructures is used. The four- 
quadrant BSE detector system consists of four separate backscatter electron 
detectors that are placed symmetrically to the sample in the sample chamber of the 
SEM. If the sample is scanned with an electron beam, different images of the 
surface structure arise at each of the detectors. This results in four different gray- 
toned images produced from the information provided by the respective detectors. 

In these images, an apparent direction of illumination can be determined, 
arising from the different shadow areas in the images. This property can be used to 
produce a 3D model of the sample from the four images. The accuracy of the 
measurements in the procedure is in the single-digit um range. 

The above approaches are not suitable for the support of nanohandling inside 
the SEM, because the resolution is not high enough, the systems are too bulky, or 
they have an impact on the handling process. 
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5.3.2 Stereoscopic Image Approaches 


A frequent area of application for the stereoscopy in the SEM is the analysis of 
surface topographies, for example in researching crack formation in materials. 

In [21], surfaces with pyramidal structures, which are used in solar technology, 
are analyzed. They use an edge detection algorithm to create a wire frame model of 
the pyramidal structures for each of the left and right views of a stereo image pair. 
Using the disparities at the lattice points, a height model can then be created. In the 
analysis of regular structures, in which there are many edges, a feature-based 
stereo algorithm for the processing of the images is used. The stereo image pair is 
created with a tilt angle of 30°. 

In [22], a combination of several procedures is used. In the first calculation 
phase, features from the images are extracted. They serve to provide the rough 
alignment and the geometrical equalization of the images. After this, the disparities 
of some of the reference points are calculated using a correlation-based 
approach. From this, a not very dense but, on the other hand, precise (accurate at 
sub-pixel level) disparity map arises. Finally, the density of the disparity map is 
increased by means of the reference points and the application of an interpolation 
procedure. 

The standard software package MeX (Version 4.1) is based on the work in [23]. 
It calculates a digital surface model (DSM), a three-dimensional height map from a 
stereoscopic image pair following a correlation-based procedure. Following this, 
the reconstructed surface can be analyzed and measured. The software has a 
graphical user interface for data input and the analysis of the results. As data is fed 
in, the mapping of the input image is given, as well as the relative tilt angle, the 
operating distance, and the pixel size of the images. The latter is used primarily for 
the calculation of absolute object coordinates. The other information enables the 
overall displacement of the images to align them against each other. One function 
allows the images to be automatically aligned in this way. 

Once all the data has been fedin, the calculation of the DSM can begin. This is 
carried out interactively in several stages. To start with, the user establishes a 
region of interest (ROI), that is, an image extract for which the calculation is to be 
carried out. Then, an “input DSM” is calculated and displayed. This is a rough- 
textured DSM in which few points of the image data can be used for calculating. 
With this snapshot, the user can work out whether the image data is usable or 
whether operation parameters need to be modified. At this point, the calculation 
can either be stopped or continued. With a continuation, a detailed DSM will be 
calculated. This can then be used for comprehensive volumetric measurements. 

With stereoscopic calculations, a correlation-based procedure is used in which 
the size of the correlation window is dynamically calculated. The image data is 
processed hierarchically using a coarse-to-fine process. With an input image size of 
1024 x 768 pixels, the disparities are determined from about 15,000 sampling 
points. From this, a wire frame structure is derived, which is used for the 
interpolation of the DSM. The reference image is superimposed as a polygon 
texture over the height map when the DSM is displayed, in order to complete the 
three-dimensional view. As an alternative, the height information can be observed 
in the form of a depth map, corresponding to a top view of the DSM. 
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The described stereo approaches have not been developed for the support of 
nanohandling. They use a standard SEM that is only equipped with a tiltable 
sample table. One disadvantage of this approach is that the handling station, which 
consists of the handling object, robot, and tools, has to be tilted as well. This can 
take several minutes and there is a high risk that the object changes its position. 
Therefore, this approach is not suitable for the generation of stereo images for the 
observation of handling processes. 

Also, images of technical handling processes are different from images of 
surfaces. The major difference in technical process images is the high number of 
low-texture regions. This leads to heavy noise in these regions as a result 
(disparity map) of the stereo algorithm. Thus, misinterpretation of the object and 
tool positions may result. 

The existing approaches do not fulfill the requirement for nanohandling inside 
the SEM. 


5.4 3D Imaging System for Nanohandling in an SEM 
5.4.1 Structure of the 3D Imaging System for SEM 


In this section, the structure of the 3D imaging system, which fulfills the require- 
ments of micro- and nanohandling, is described [24-26]. The system is 
schematically represented in Figure 5.13 and consists of five main components. 

Because the 3D imaging system has to be suitable for observing manipulation 
processes, the technique of beam tilting is used. To overcome the known problems 
of handling in the beam-tilting system, the system described below was developed. 

This system does not need a special electron column and can be flexibly and 
universally applied in each standard SEM. The stereo images are acquired through 
the beam-tilting system which is controlled by the beam-control unit. The images 
are then suitable for the following processing work. Before the images are 
transferred into the 3D module, they are pre-processed and filtered by standard 
methods such as the median filter. Noise levels in the images are therefore reduced 
and a contrast enhancement is carried out. 

After the pre-processing, the images are sent to the 3D module. This consists of 
two sub-modules: the vergence system and the stereo system. The vergence 
system is necessary to line up the images against each other and to compensate for 
unwanted shifts, rotations, and different zoom scales, where it is necessary. In this 
way, prepared images are sent to the stereo system which processes the image data 
and provides the 3D information (disparity maps). These results are then processed 
in the last module (Figure 5.13) which provides the data for the graphical user 
interface (GUI) or a robot control system. 

The image acquisition system and the 3D module are described in more detail 
in the following sections. 
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Figure 5.13. Structure of the 3D imaging system 
5.4.2 Image Acquisition and Beam Control 


Beam-tilting is a suitable stereo image generation technique for nanohandling tasks 
in the SEM. Until now, however, this technique has only been available in SEMs 
that have a special electron beam column [10-12], and it cannot be used in 
standard SEMs. Therefore, the described system provides an external and flexible 
lens system which can be installed in every SEM [24]. 

For the beam-tilting system a magnetic lens is used. This is fixed under the 
electron gun with the help of a metal arm. The arm can be flexibly mounted on the 
side panel of the SEM and the position of the lens can be adjusted. Additionally, 
the lens is supplied with power via a vacuum-suited cable entry. 

The task of the external lens system is to deflect and tilt the beam twice. With 
this procedure, two images from different perspectives of a sample are generated 
by deflected electron beams. By using the integrated beam-shift unit of the SEM, 
the beam is deflected in one direction, and with the additional external lens system 
the beam is then deflected in the opposite direction. In Figure 5.14, the principle is 
shown schematically. With this approach a tilt angle of max. 3° can be achieved. 

In order to generate two different perspectives, a beam control unit is needed. 
This unit is schematically represented in Figure 5.15 and consists of four main 
components. 

With the integrated SEM beam-shift unit the beam entry position can be 
changed with reference to the center position of the lens. The image acquisition 
unit acquires two images, which have to be generated from two different 
perspectives. This means that the view direction has to be changed. Thus, the beam 
has to be shifted in the right position and the polarity of the lens must be changed 
as well. This is carried out by the control system, which sends the necessary 
control commands to the electronic polarity unit and to the beam shift unit. 
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Figure 5.15. Structure of the beam control 


3D Imaging System forSEM 151 


5.4.3 The 3D Module 


The 3D module is a very important part of the 3D imaging system. Its purpose is to 
extract 3D information from stereoscopic images. It consists of two components, 
the vergence system and the stereo system (Figure 5.16). The functionality and the 
algorithms of these systems are described below. 
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Figure 5.16. Flowchart of the 3D module 























The disparity calculations and therefore the acquisition of the 3D information 
are carried out in the stereo system. This receives a stereoscopic image pair which 
is processed over two layers. The estimation layer calculates disparity estimates, 
and the coherence layer determines stable disparity values and presents them in the 
form of a disparity map. Apart from this, other results, which are described in the 
next section, can also be calculated. Because of the additional special image 
acquisition in an SEM, the stereo images are potentially distorted. Vertical shifts, 
counter-rotation, or different zoom scales could result between the two stereo 
images. This makes it difficult for the stereo system to extract the image 
disparities. However, it is possible to compensate these distortions within limits by 
a vergence system, so that the images can be properly processed by the stereo 
system. 

The stereo system then receives its input from the vergence system, which 
aligns the stereo image pair. The vergence system can be passed through a number 
of times. In each pass, a fixation point is chosen and the image pair is aligned 
according to the vergence signals (shift parameters). The aligned images are sent to 
the stereo system to calculate the disparities relative to the chosen fixation point. 
The results of each pass are accumulated, which provides robust 3D information. 

In the next section the stereo system is described first, because the results of 
this system are needed for the vergence system, which will be described 
afterwards. 
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5.4.3.1 Stereo System 
The presented stereo algorithm consists of two processing layers, the estimation 
layer and the coherence layer (Figure 5.17). 
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Figure 5.17. Structure of the stereo algorithm 


In the estimation layer, a number of disparity estimations for every pixel are 
calculated. This leads to approximately correct disparity values but also to random 
and incorrect values. Therefore, the coherence layer has the task of separating the 
correct from the incorrect estimations and calculating a stable result. The central 
data structure of the algorithm is the estimation cube. It is created by the estimation 
layer and analyzed by the coherence layer for disparity detection. Both the 
estimation and the coherence layers are described in more detail in the following 
sections. The results of the stereo algorithm are a sharp and high-density disparity 
map, which gives the relative depth, a validation map, which gives the reliability of 
the disparity values, and a texture map showing the regions with low or high 
texture in the images. The validation and texture values are needed for the 
vergence system which is explained below. 

The task of the estimation layer is to estimate several disparities for all image 
pixels. The base unit of this approach is a disparity estimation unit (DEU). This 
unit is capable of estimating the disparity down to one pixel position. The 
estimation cube consists of lots of such units. 

Figure 5.18 schematically shows a DEU (bottom of the figure). Such a DEU is 
built up of several neuronal layers, which are based on the concept of the energy 
model described in [3, 27, 28]. The two major layers are the simple cell layer and 
the complex cell layer. The receptive field of a simple cell is modeled by Gabor [3, 
28, 29]. 

The discrete filter coefficients are normalized to eliminate the DC component. 
The response of one simple cell is given by: 


3D Imaging System for SEM 153 


s(x, y) = >» g(i;0) L (x+i, y)+ g@0)-L,.(x+i, y). (5.9) 


i=-n 


This means that a convolution between an image scan line and the coefficients of 
the Gabor function is carried out. The complex cells are modeled as a quadratic 
filter and the receptive fields are calculated by the squared input from at least two 
simple cells. 
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Figure 5.18. Structure of a disparity estimation unit (DEU): the simple cells of c_ and c, are 
not shown 


With these complex cells, disparity estimations can be carried out. However, it 
is known that the response of complex cells depends on the local contrast. To avoid 
this undesirable effect, a solution described in [27] is used. For the disparity 
estimation, three complex cells are used and the responses are compared. The 
complex cells differ in their parameters. The first complex cell (c+) detects slightly 
positive disparities, the second detects slightly negative ones (c.) and the third 
detects zero disparities (co). The difference between c+ and c. is normalized by cp. 
This approach covers only a small disparity range. 

Since the detection of larger disparities is desirable, a combination of the phase 
shift and the position shift model [4, 5, 16, 27, 30] is used. This leads to the 
response of one DEU by 


d(x y;s) = c, (x, y38,+) —c_(X,y38,-) 
í Co (x, 38,0) 





S, (5.10) 


where 
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The parameter s determines the tuned disparity of one DEU and @is fixed to 7/4. 
Because of the fact that (c,-c/c,) gives the relative disparity to s, s must be 
subtracted from the result. A group of DEUs with different s parameters can cover 
a bigger disparity space than a single DEU. If the number of DEUs increases, then 
the detectable disparities will increase as well. Values for s from —4 to 4 at the step 
size of 1/3 are suitable for most cases. 

The estimation cube (Figure 5.17 right) is a three-dimensional structure which 
consists of numerous DEUs. The size of the cube in the x-y-direction is the same 
size as the input images. This is because the disparities for all pixels of the images 
are estimated. The size of the third dimension depends on the disparity search 
space, that is, how many DEUs are necessary to cover a desired disparity search 
space. 

The coherence layer analyzes the results of the estimation layer which are 
stored in the estimation cube. The result of one DEU can either approximately 
represent the correct disparity or can have a more or less random value. The latter 
case will occur if the true disparity lies outside the working range of the DEU. 
However, the estimation cube consists of several DEUs with overlapping working 
ranges for one image position. So, the disparity search space can be increased and 
a robust estimation can be carried out. 

To find the correct disparity for one image position (x,y), all relevant DEUs are 
grouped to a disparity stack, according to [3, 28]. Using coherence detection, it is 
possible to estimate the correct disparity, so the biggest cluster of DEUs which has 
similar disparity estimations is calculated. The final disparity of a position (x,y) is 
calculated from the average of the estimated results of the coherence cluster: 


alt 


Dis(x, y) = ic 


Xda, yS), (5.14) 
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where C,, is the set of DEUs in the coherence cluster of the disparity stack at 
position (x, y) and S(i) is a mapping function mapping / to the tuned disparities s of 
the DEU. 


d 








a 





Figure 5.19. Search for the coherence cluster: a. typical curve in a disparity stack; b. first 
group of sorted DEUs; c. biggest cluster 


In every disparity stack, the biggest DEU group with similar disparity values 
is searched (Figure 5.19). In Figure 5.19a, a typical curve in a disparity stack is 
shown. The DEUs are shown on the abscissa and the calculated disparities are 
shown on the ordinate. 

In the first step, the DEUs are sorted by their disparity values in ascending 
order. Let the disparity values of the sorted DEUs be D(i) (i€],...,N), with N 
being the size of the disparity stack. Then N groups of DEUs (Gj,...,Gy) are built 
up and determined by: 


G; = {D(),...D(k)} (5.15) 
k; = max( {k| i<k< Na D(k)- D(i)< €}). (5.16) 





Figure 5.19b shows the first group of sorted DEUs. Let d(G,) = D(k;)— D(i), then 
the group consists of as many consecutive DEUs as possible, where d(G;) is 
maximal and is not greater than e. The biggest cluster found in this way is the 
coherence cluster C,, of the disparity stack at the position (x, y). Figure 5.19c 
shows the biggest cluster for the example. In the worst case, several groups with 
the same size are found. In this case, the group is chosen in which d(G;) is 
minimal, as a large group with a small range of disparity values indicates a stable 
estimation. 

The parameter ¢ should be carefully chosen. If € is too low, only small groups 
are found, and the likelihood for clusters of the same size is increased, which in 
turn leads to ambiguities. If ¢ is too high, the average of the disparity result is less 
precise, because noise coding DEUs start to appear in the coherence cluster. 
Additionally, the calculation time increases. It was found experimentally that a 
value of ¢ = 0.5 is a good compromise for SEM images. 

One major problem of the generated images is that in most cases the probability 
of regions having low or no texture is high. In these regions (e.g., a black 
background) only small changes in the intensity exist. Therefore, no disparity can 
be detected by simple cells and the disparity map is noisy in these regions. In this 
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chapter, a solution using a dynamic texture-based filter for overcoming this 
problem is described. One important part of the filter is the texture map. This 
shows regions with low and high texture of the images. Using this information, a 
dynamic texture-based filter can be built up, which reduces the noise of the 
disparity map. To create the texture map, the complex cells co(x, y; s, 0) of all 
DEUs have to be used, since they detect the local texture in the stereo image pair. 
The final result is determined by averaging all complex cell cy responses of a 
disparity stack. Therefore, the texture of an image position (x, y) is given by: 


N 
Tey) = 59 cole 755(0.0), (5.17) 
i=l 


where S(i) as above is a mapping function for mapping the index i to the tuned 
disparity s of the DEU and N is the size of the disparity stack. If the texture map is 
calculated for every position (x,y), then each disparity value will have a 
corresponding texture value. In order to reduce the noise in the disparity map, all 
regions with a texture value which is smaller than a threshold are taken out. 
Because the range of texture values depends on the image data, a fixed 
threshold is not suitable. A dynamic adaptation of the threshold is therefore 
proposed. A solution can be found by analyzing the histogram of the texture map. 
Figure 5.20a shows a typical histogram of an image pair with several textureless 
regions. It can be seen that most of the values are found in the left part of the 
texture domain which is limited by the smallest min({T(, y)}) and the biggest 
max({T(x, y)}) value. In contrast, Figure 5.20b shows a histogram of an image pair 
with few low-texture regions. Here, most texture values are located in the middle. 
These histograms lead to the conclusion that a threshold close to the dotted line is 
preferable. With such a threshold, large regions of the disparity map can be filtered 
if the noise level is high, and on the other hand it is possible to filter only small 
regions of the map if the noise level is low. The threshold can be calculated by: 


bhres = (max({7 (x, yy) T min({7 (x, y}) “Pp 


5.18 
+ min T(x, y)}). oe) 


where all disparity values T(x, y) < tyes are ignored and filtered. The parameter p 
determines the portion of the filtered texture values. A value of p = 0.05 yields 
good results in most cases for images showing handling processes. 


5.4.3.2 Vergence System 

Because of the special acquisition of the stereo images in the SEM, the images 
might differ in how well they can be used for processing. With a high-quality 
stereo image pair, only horizontal disparities occur, meaning that image regions are 
shifted either to the left or to the right (or not shifted at all in the case of zero 
disparities). However, inaccuracies during image acquisition could lead to 
additional vertical disparities, which allow image regions also to be shifted up or 
down. Figure 5.21 shows three examples. 
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Figure 5.20. Typical texture histograms: a. many low-texture regions; b. few low-texture 
regions 








c 


Figure 5.21. Possible distortions in the images: a. vertical shifts; b. rotation; c. different 
zoom 


If the SEM beam does not hit the specimen on exactly the same spot after 
changing the viewing angle, the resulting images will have a global vertical shift 
(Figure 5.2la). A counter rotation of the images will lead to vertical disparities, 
particularly in the left and right areas of the images (Figure 5.21b). If the images 
differ in a slight zoom scale, vertical disparities will occur in the top and bottom 
areas of the images (Figure 5.21c). 
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The first case is the easiest to compensate, because the vertical disparities are 
constant over all image regions. The other two cases are more difficult because the 
vertical disparities differ from region to region. 

The occurrence of vertical shifts will strongly affect the results of the stereo 
system, as it can only detect horizontal disparities. During image processing, it is 
not known if and where vertical disparities are present. Therefore, a pre-processing 
of the images, which compensates, for example, for rotational differences, is not 
possible. 

The concept of a vergence system as described in [24, 31] can be used to 
reduce these problems by aligning the images not only horizontally, but also 
vertically. So, vertical disparities are taken out. Images with different vertical 
disparities in different image regions can be processed in multiple passes using 
appropriate vertical shifts in each pass. The validation values of the coherence 
layer indicate the image regions where the disparity calculations have been 
successful. Therefore, after each pass, the disparities that have higher validation 
values than the disparities calculated in prior passes are accumulated. 

The vergence system is used as an enhancement of the stereo system. The 
stereo system is not altered in its functionality, but used multiple times with 
different input parameters. Before each call, a fixation point is chosen. The 
fixation point selection is carried out using the validation and texture map. 
Vergence signals are generated accordingly, that is, horizontal and vertical shift 
parameters to fuse the input images at the fixation point. The aligned images are 
sent to the stereo system for calculation and the results are accumulated. 


5.5 Application of the 3D Imaging System 


In this section, the results of the 3D imaging system are first presented and 
discussed. Afterwards, two applications of the imaging system, the handling of 
CNTs and the handling of crystals, are shown and discussed. 


5.5.1 Results of the 3D Imaging System 


Figure 5.22a,b shows the input images and Figure 5.22c—i the results of the stereo 
algorithm. The tilt angle is small (about 2°), so only small disparities are 
detectable. The images show a gripper [32] with a jaw size of about 2 um and a 
nanotube (produced by Engineering Department at Cambridge University) of about 
300 nm diameter and 12 um length seen from the side with a magnification of 
3000 times. The texture around the objects is low and there is a contrast difference 
between the images as well. The calculated disparity map (Figure 5.22f) shows 
dark regions for large distances and bright regions for small distances with 
reference to the observer. In Figure 5.22g a 3D plot of the disparity map is shown. 

The stereo algorithm calculates a high-density disparity map in sub-pixel 
accuracy. The smallest step size of the gripping process in the direction of the 
nanotube was | um which could be detected by the approach. 

It is also noteworthy that the nanotube in the disparity map is displayed wider 
than in the input images. This is due to the size of the receptive fields of the DEUs. 
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Figure 5.22. Input images and the resulting images of the stereo algorithm for one nanotube 
and a gripper: a. and b. input images; c. cyclopean view; d. validation map; e. texture map; f. 
unfiltered disparity map; g. 3D plot of the unfiltered disparity map; h. filtered disparity map; 
i. 3D plot of the filtered disparity map 


This effect can be reduced by increasing the image resolution (and possibly by 
defining a region of interest) or reducing the size of receptive fields. 

It should be noted that, because of the low-texture regions around the gripper 
and the nanotube, noise arises around it in the disparity map (Figure 5.22f) and the 
tool and the tube are not clearly seen in the 3D plot in Figure 5.22g. This is a 
common problem for all stereo algorithms, since the corresponding problem cannot 
be solved in textureless regions. However, the described stereo algorithm provides 
a texture map (Figure 5.22e) which shows regions of high and low texture in the 
input images. Therefore, with this map, a dynamic threshold can be calculated and 
used to filter the noisy disparity map. The result of the filtering is shown in Figure 
5.22h and the 3D plot is shown in Figure 5.221. In the result image and the 3D plot, 
the noisy regions of the disparity map are removed and the contours are thinned by 
means of the determined cyclopean view (Figure 5.22c). Therefore the objects are 
clearly shown and only the important regions are available. This result is more 
suitable for the further processing steps than the noisy disparity map. Because of 
the textureless region inside the gripper, only the contour is shown in the disparity 
map. 
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5.5.2 Application for the Handling of CNTs 


As previously referred to, the manipulation of carbon nanotubes has great impor- 
tance in nanotechnology. There is a big advantage in the use of an SEM for 
observing manipulation processes. The manipulation task consists of gripping the 
nanotube in a controlled way and placing it on a TEM (transmission electron 
microscope) grid (Chapter 2). The procedure is teleoperated or carried out auto- 
matically. 

For this purpose, multiwalled carbon nanotubes (MWCNT) are used. These are 
produced on a silicon substrate through the chemical vapor deposition (CVD) 
procedure. The availability of these multiwalled tubes was facilitated through the 
Engineering Department of Cambridge University, UK [33]. 

The main difficulty here is to bring the gripping tool and the nanotubes to the 
same level. Only in this way is it possible to achieve a precise gripping of the tube. 
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Figure 5.23. Left: different views from the gripper, which is under the nanotube: Right: 
different views from the gripper at the same level as the nanotube. 
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Through the 3D imaging system, continuous stereo images of the manipulation 
process are produced. These are displayed on the GUI as an anaglyph image or 
they can be shown as a 3D plot. The anaglyph image can be used by the operator, 
with the help of suitable red—green glasses, to produce a stereoscopic impression of 
the manipulation procedure. The 3D plot displays the results of the stereo algo- 
rithm. These results can also be used by the operator to control the manipulation 
procedure. Furthermore, the underlying results can be used by a robot-controlled 
system to determine the corresponding positions of object and gripping tool. This 
is indispensable if an automatic manipulation is to be carried out. In Figure 5.23, 
the start and end positions of an approach by the gripper moving up towards the 
nanotubes are shown. Here, several different 3D views of the gripper and the tube 
at the start of the approach are seen in the left image area. It can be seen that the 
nanotube is lying at an angle (about 45°) and the gripper is below it. On the right 
side of Figure 5.23, more 3D views are shown. Here, the end position of the 
approach can be seen. This shows that the gripper jaws are positioned at the same 
height as the upper part of the nanotube. 

After the gripper has been set up, the gripping process can begin. A voltage is 
applied to it in order to close the jaws. This way, the nanotube is gripped. Now, the 
nanopositioning unit can be moved and the position of the TEM network can be 
controlled. 


5.5.3 Application for the Handling of Crystals 


A further manipulation process involves the sorting of crystals. In this process, the 
crystals are moved with the help of a so-called STM tip. The difficulty here is that 
in the manipulation process, the tip has to lie against an outer wall of the crystal, 
otherwise the crystal cannot be moved. If the tip is too high, it misses the crystal. If 
it is too low and touches the base, it may become damaged and unusable. So the 
operator needs 3D information in this task, too. Similar to the process for 
manipulating nanotubes, with the help of a 3D imaging system, an anaglyph image 
and a corresponding 3D plot are available to the operator. This makes it possible 
for the operator to control the tip and manipulate the crystal. Figure 5.24 shows a 
stereo image pair and a 3D plot of such a manipulation process. The crystals have a 
size of about 20 um and the STM tip shown has a diameter of about 2 um. It can 
be seen here that the tip is positioned under the crystal. The crystal can therefore be 
moved and sorted by the tip. 


5.6 Conclusions 
5.6.1 Summary 


In this chapter, various possibilities for obtaining 3D information from SEMs with 
respect to nanohandling are discussed. 

For this, basic concepts such as the principle of stereoscopic image approaches 
in the SEM, mathematical basic and biological vision systems are presented. In 
addition, current systems used to obtain 3D information using SEMs are presented 
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and discussed. These systems have major disadvantages in reference to nano- 
handling actions. Therefore, a universally applicable 3D imaging system that 
fulfills the requirements in nanohandling is discussed in more detail. In this system, 
stereo images are generated by beam tilting followed by processing of the image 
data. One major part, the 3D module which consists of a vergence and a stereo 
system, is depicted in detail. 
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Figure 5.24. a. input images and b. the 3D plot of the manipulation of crystals by an STM 
tip 
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The stereo system is based on a biologically motivated energy model with 
coherence detection. By means of the used stereo system, a sharp and high-density 
disparity map in sub-pixel accuracy can be calculated. Also, a dynamic filter is 
shown which gives the stereo system the capability to detect regions of low texture 
in the input images and thus to remove the noise of the disparity map in these 
regions. The vergence system with an enhanced solution for fixation point selec- 
tion is presented. This enables the 3D module to supply accurate and robust results, 
even with disturbed input images (rotated, vertically shifted, or with different zoom 
scales). 

The applicability of the 3D imaging system is shown by the handling task of 
nanoobjects such as nanotubes and crystals. 


5.6.2 Outlook 


In future work, the calculation time of the 3D imaging system will be further 
decreased. For this, an optimized implementation in hardware (e.g., on a field 
programmable gate array, FPGA) for the 3D module will be developed. Therefore, 
it will be necessary to investigate some processing strategies for the hardware 
design. For example, a suitable partitioning of the vergence and stereo system has 
to be found. 

The 3D imaging system will be able to calculate only a certain region of 
interest, so that the interesting parts (e.g., the region between gripper and nanotube) 
can be observed in more detail. 
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6.1 Introduction 


The development of microrobots for nanohandling, with ever-growing demands 
regarding flexibility and automation, raises the problem of appropriate process 
control. With dimensions of the parts to be handled of sometimes considerably less 
than 100 nm and with a typical positioning accuracy in the nanometer range, nano- 
handling applications have detached several orders of magnitude from the 
operators’ macroscopic realm of experience. Powerful sensory feedback is required 
to overcome this scale gap [1] and to be able to handle and manufacture such small 
devices. The extraction and transfer of process information from the micro- and 
nanoworld into the macroworld are great challenges. However, they are important 
preconditions for closed-loop microrobotic control systems and thus a key to 
reliable nanohandling [2]. 

Obviously, visual process monitoring using different kinds of microscopes is 
essential (Chapters 2, 4, and 5). Apart from image processing, force feedback is the 
most important sensor application in nanohandling [3-5]. The main challenge is 
the control of the interaction between the part and the microgripper or end-effector 
[6]. A lot of micro- and nanoscale parts are very sensitive to gripping or contact 
forces due to their brittle materials, filigree structures, or small gripping areas. 
They can easily be damaged or even destroyed [3]. The integration of force sensors 
into grippers and end-effectors provides essential information on the nanohandling 
process for both the operator working with telemanipulation devices and the 
control system operating in automatic mode. 

In the first part of this chapter, an overview of the fundamentals and principles 
of micro/nano force measurement will be given, with an emphasis on the special 
requirements in force feedback for nanohandling by microrobots. The state-of-the- 
art in force sensing using robot- and atomic force microscopy (AFM) based 
nanohandling systems is presented in the second part of this chapter. 
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6.2 Fundamentals of Micro/Nano Force Measurement 


One basic principle for the measurement of a force is to compare it with a 
previously known force. This has been done for several thousands of years and, 
nevertheless, is still an important topic of current research activities [7]. Classical 
mechanics provides the corresponding theory and addresses the interaction 
between macroscopic objects. It is based upon the research and theory developed 
by Sir Isaac Newton (1643-1727). 


Newton’s Second Law. If an external force acts on a body, the body is 
accelerated in the direction of the force. Thus force can be defined as a vector. 
Isaac Newton discovered that the acceleration a is proportional to the force F and 
inversely proportional to the mass of the body m, a scalar: 


(6.1) 


S| 


This equation is known as Newton’s Second Law. It enables the definition of 
mechanical units. According to the SI system, mass (kg), length (m) and time (s) 
are the basic units. Acceleration and force are units derived from the basic units. 
The unit of force is the newton (N). One newton is defined as the force which 
accelerates a mass of 1 kg with 1 m/s’: 


m 
1 N=1 kexs- (6.2) 


In the following, the main principles of force measurement are presented and 
the requirements for force feedback in nanohandling are described. 


6.2.1 Principles of Force Measurement 


Altogether, there are six principles of force measurement, which can be classified 
as follows [8]: 


1. Comparison of the unknown force with the force of gravity of a known 
mass. 

2. Measurement of the acceleration of a known mass on which the unknown 
force acts. 


3. Comparison of the force with an electromagnetically generated force. 

4. Conversion of the force into pressure (of a liquid) and measurement of the 
pressure. 

5. Direct measurement of the (absolute) deformation of an elastic body caused 
by the force. 


6. Measurement of the (relative) strain in an elastic body caused by the force. 
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The last principle mentioned is the one mostly used for force sensors. Also, the 
fifth principle and — with some limitations — principles 3 and 4 are sometimes used 
for force measurement. The measurement of a force by measuring the acceleration 
of a known mass is not of practical importance for nanohandling. 

In a physical sense, force is a vector which acts on a point. When measuring a 
force, its vector has to be transformed into a scalar. In practice, the force to be 
measured cannot act on a body directly as materials can only withstand limited 
mechanical stress. To be exact, there is no force measurement technique but rather 
a technique to measure mechanical stress fields which are caused by an external 
force (principle 6). In the following, this will be explained in more detail. 

A force acting on a fixed body will deform this body and cause strain and 
mechanical stress. If the stress o is equally distributed, it can be calculated dividing 
the force F by the cross-section of the body A: 


cee (6.3) 


The relative elongation or strain g is defined as elongation A/ caused by the 
force F divided by the length of the body / in the direction of the force: 


é=—. (6.4) 


The mechanical stress and thus the required force is determined from strain 
measurement. Hooke’s Law provides the basis for this. 


Hooke’s Law. In the case of a uniaxial stress condition, within the proportional 
area of the stress-strain curve, Hooke’s Law applies for a uniaxially stressed beam: 


o=E-e. (6.5) 


Young’s modulus E corresponds to the slope of the straight line of the stress-strain 
curve up to the proportional limit. It is obvious that force sensors may only operate 
within this elastic area to measure reliably. 

If the strain of a body is measured for instance with strain gages, the force 
acting on it can be calculated as: 


F=e-E-A. (6.6) 


If the amplitude of a force is determined by directly measuring the deformation 
x of a force-sensitive element with known stiffness k (principle 5), as for example 
in case of the deflection of an AFM cantilever, the force F can be calculated 
according to: 


F= ra (6.7) 
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To put it simply, for the development of a force sensor a suitable physical principle 
for the measurement of either the mechanical stress o (Equation 6.3) or the strain € 
(Equation 6.6) or the deformation x (Equation 6.7) has to be chosen and technically 
realized. The parameters Young’s modulus £, cross-section A, and stiffness k of 
the sensor are predefined by its material properties and geometry. 


6.2.2 Types of Forces in Robotics 


In robotics, force sensors are used to measure the interaction forces between 
gripper/end-effector, the part to be handled, and the environment. Typically, forces 
are divided into two classes: When gripping a part, the gripping forces which 
occur between gripper and part are of special interest. When manipulating a part, 
e.g., during assembly, moving on a surface, or mechanical characterization, the 
contact forces have to be known. Sensors to measure the gripping forces are 
integrated into the gripper — ideally into the gripper jaws. Contact force sensors can 
be mounted between gripper/end-effector and manipulator (Figure 6.1). 









robot arm / 
manipulator 


gripper 


contact force 
sensor 


gripping force 
sensor 








Figure 6.1. Sensors for gripping and contact force measurement integrated into the gripper 
and the manipulator, respectively 


6.2.2.1 Gripping Forces 

A gripper has to ensure a fixed position and orientation, i.e., pose of the part with 
respect to the robot’s last joint. In this manner, the gripper has to exert and 
withstand forces and torques. The main disturbing factors are forces of inertia due 
to the robot’s acceleration as well as contact forces between the part and other 
objects within the working space. In micro- and nanohandling, forces of inertia 
play only a minor role, due to much smaller weights of the parts with respect to 
their gripping area, whereas adhesion forces prevail. However, it has to be ensured 
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that the microgripper grips the parts well enough to prevent slipping, while at the 
same time not damaging them. 

The gripping force acts from the gripper on the gripped part. It has to 
counteract a total load resulting from several single forces and torques, including 
static holding forces as well as dynamic and process-related loads. The required 
magnitude of the gripping force depends on the geometries of gripper and part as 
well as — in the case of force-fit gripping — the friction coefficient between gripper 
jaw and part. 


gripper 

















G 
Figure 6.2. Forces acting on a gripped part 


The forces occurring during force-fit gripping are shown in Figure 6.2. The 
gripping force Fg acts on the part as normal force and, thus, generates the friction 
force Fr. According to Coulomb’s Law, the friction force acts against the direction 
of motion and, thus, against the weight G of the part. If the gripped part is not 
moving, the following equation applies: 


G=F, =n-y-F;, (6.8) 


with the static friction coefficient u between gripper jaw and part and the number 
of gripper jaws n. Resolving this equation for the required gripping Fg force leads 
to: 


oq! 


F,=— 4, (6.9) 


n: 





T 


with the mass m of the part and the gravity g. When determining the required 
gripping force, it has to be considered that, on the one hand, the gripping force has 
to be high enough to compensate for the weight of the part and firmly grip it, and, 
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on the other hand, the surface pressure during the gripping must not exceed the 
maximum tolerable value for the material of the part. This problem can be 
overcome by, for instance, integrating gripping force sensors and implementing a 
suitable gripping force control. 


6.2.2.2 Contact Forces 

Forces occurring between (i) the gripped part or the gripper and the environment or 
(11) between the part and the end-effector are called contact forces. If two parts 
have to be assembled, or, for instance, the membrane of a biological cell has to be 
penetrated with a pipette, their relative position and orientation have to be known. 
Uncertainties regarding this information occur mainly due to measurement errors, 
robot inaccuracies, and disturbing external influences. Therefore, the assembly can 
fail or parts can even be damaged by assembly forces being too high. 

By measuring contact forces, damage can be prevented, e.g., during an unfore- 
seen contact due to positioning inaccuracies, as well as gaining additional infor- 
mation about the relative position of the part and its join partner. Contact force 
measurement can not only complement gripping force measurement, but also 
provide additional information for the robot control and the operator, which may 
complement or even supersede image processing. A typical example of the poten- 
tial benefit of contact force measurement is the so-called peg-in-hole benchmark. 


6.2.3 Characteristics of the Micro- and Nanoworld 


In principle, force measurement techniques known from the macroworld can be 
used in the micro- and nanoworld as well. However, there are a few characteristics 
and constraints to be taken into account, which can limit the usability of these 
techniques at small scales. 

Nanohandling differs clearly from classical, i.e., macroscopic, handling and 
assembly because of different scales; four different scaling areas can be distin- 
guished. Table 6.1 gives an overview of these areas, together with suitable actu- 
ators and force sensors. The definition of the different ranges is not to be taken 
strictly but rather to give an indication. Regarding the object size and the accuracy 
required for positioning and handling, two different areas have been introduced 
because the accuracy often has to be better than the object size by a factor of 100 to 
1000. 

Handling and assembly in the macroscopic domain, usually carried out using 
industrial robots, is state-of-the-art today. This area is dominated by friction and 
inertia forces which lead to a predictable behavior of the parts. Numerous handling 
strategies for force- and form-fit gripping as well as techniques for a systematic 
gripper design have already been developed [9]. 

The situation changes during handling of parts with dimensions smaller than 
1 mm and with a mass smaller than 1 mg. Due to an increasing ratio between 
surface area and volume, forces proportional to the surface area increase compared 
to forces proportional to the volume of the object (Figure 6.3). Friction and inertia 
forces dominating during macroscopic manipulation decrease compared to surface 
forces, and the parts tend to either stick to the end-effector or show a behavior 
difficult to predict, like jumping. Among the prominent surface or adhesion forces 
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are surface tension, electrostatic forces, and van der Waals forces. They have been 
described in detail in the literature [10, 11]. 


Table 6.1. Scale domains in handling and assembly 




















Scale Objectsize Accuracy Dominating Actuators Force sensors 
forces 
Macro >1mm > 10 um gravity, friction micrometer state-of-the- 
force screws, art, 
electric commercially 
motors available 
Meso 0.1-1 mm < 10 um friction and micrometer partly state- 
surface forces screws, of-the-art, 
electric forces in mN 
motors, solid to uN range 
state actuators 
(e.g., slip- 
stick) 
Micro 1-100 um <100nm surface forces solid state difficult to 
actuators develop, 
(e.g., slip- forces partly 
stick) in uN range 
Nano < 100 nm < 10 nm surface and solid state difficult to 
atomic forces actuators develop, 
(e.g., flexible forces partly 
hinges) in nN range, 
AFM 
technologies 
required 
adhesive forces gravity 


force 






critical 
range! 
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part size 
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Figure 6.3. Scaling effects caused by surface and volume forces 
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Due to the dominating surface forces, the concepts and techniques for the 
handling and development of grippers known from the macroworld cannot be 
applied without modifications. One of the challenges in nanohandling consists in 
the modeling and reduction or, even better, in the utilization of surface forces. 
Different approaches have already been investigated [12, 13], but their practical 
use is still limited. A systematic method for the control of adhesion forces does not 
exist. 

Possible ways to reduce adhesion forces include amongst others: 


e adaptive design to minimize the contact area between gripper/end-effector 
and object; 

e use of “rough” surfaces within the contact area; 

e reduction of gripping forces by using sensors to measure gripping forces; 

e use of hard materials in the contact area, especially for the gripper/ 
end-effector; 

e reduction of surface tension in the vacuum, e.g., in the scanning electron 
microscope (SEM), because of reduced humidity; 

e minimization of electrostatic attractive forces by using electrically con- 
ducting and grounded grippers and objects. 


It is the goal of the first four methods to reduce the actual surface area and thus 
limit the most important factor for surface forces. However, at the same time, these 
methods lead to an increased surface pressure between gripper/end-effector and 
object. The methods to be used are always a compromise between small adhesion 
forces and a load which is still tolerable for the part. Handling in a scanning 
electron microscope has the advantage of reduced surface tension due to a lower 
humidity in vacuum; at the same time, problems can occur because of charging 
effects and electrostatic forces. 

Handling in the micro- and nanoworld has further characteristics and con- 
straints which are, however, not directly related to the development of sensors for 
force feedback for nanohandling. These include, e.g., high accuracy requirements 
for positioning, the ability to handle a large variety of geometries and parts 
materials, sensitivity against environmental influences, three-dimensional 
assembly, etc. These characteristics have already been discussed in detail in the 
literature [4, 14, 15]. 


6.2.4 Requirements on Force Feedback for Nanohandling 


From the constraints in the micro- and nanoworld described above, specific 
requirements arise for force feedback during nanohandling. As existing commer- 
cial sensors cannot be applied, the use of novel microgrippers and end-effectors 
requires the development of special micro force sensors, which take the following 
aspects into account: 

Limitation of surface pressure. Small gripping areas and small tolerable 
surface pressures require high measurement resolution and a low response 
threshold, i.e., minimum detectable force. To limit the surface pressure during the 
first contact between gripper/end-effector and object, a force measurement system 
with high dynamics and real-time capability is required. Force measurement using 
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image processing, as could be used in a scanning electron microscope, is often not 
precise enough and especially is not real-time capable. 

Design of microgrippers. The development of micro force sensors is affected 
by a set of additional parameters. The accuracy which can be achieved by a 
specific gripping force depends on the positioning accuracy of the gripper jaws or 
end-effectors. To achieve maximum accuracy, the gripper’s transmission system, 
which transfers the actuators’ motion into a movement of the gripper jaws, has to 
be free from backlash. Another constraint results from the size of the objects to be 
handled. Micro force sensors with very small dimensions are required because of 
the dimensions of the grippers/end-effectors. The sensor itself has to be within the 
same range. 

Integration of sensors. Ideally, micro force sensor and gripper should be 
connected in such a way as to enable a simple exchange of either the gripper jaws 
or the sensor. At the same time, a safe, force-fit, and perhaps even form-fit 
connection between gripper and sensor has to be ensured. The problem here is that 
the sensor should be placed close to the point of force transmission and a certain 
modularity has to be preserved — such as the exchangeability of end-effectors/jaws, 
a low number of connecting cables, etc. These factors have to be considered during 
the first development phase of the micro force sensor and microgripper. Otherwise, 
the integration of the sensor could become very difficult, and the risk of damage to 
the mechanically sensitive micro sensor could increase. 

As already mentioned, micro force sensors can either be used as separate, 
mountable sensors or they can be directly integrated into the force-sensitive 
element of the gripper. The advantage of the first solution is a universal force 
sensor that can be used for different grippers/end-effectors and within different 
applications. This can often be found in case of industrial robots but it is not yet 
available for nanohandling. In the case of the second solution, an example would 
be strain gages that are glued onto the end-effectors, or force-sensitive material 
which is deposited on the microgripper. This setup solves the problem of a possibly 
reduced stiffness as in case of a separate sensor. However, the usability of these 
techniques largely depends on the design of the microgripper and is not as 
modular. 

Use in clean rooms and scanning electron microscopes. Further constraints 
have to be taken into consideration when the micro force sensors are used within a 
clean room. During manufacturing of micro- and nanostructures, a production 
environment with constant temperature and humidity is required to reduce the 
number and the size of defects. Particles like dust can influence the functionality of 
these structures. Hence, robots, grippers, and sensors have to be clean room 
compatible and should produce almost no particles by abrasion or the like. 

When using the micro force sensors inside a scanning electron microscope, 
particular conditions imposed by the vacuum and the electron beam have to be 
considered. These will be described in more detail in Section 6.2.5. 

Technical specifications. Although nanohandling imposes several constraints 
on the development of micro force sensors, the technical specifications of the 
sensors cannot be disregarded. Measurement ranges between several 100 mN down 
to a few micronewtons are required. The resolution should be in the range of 
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several tens of micronewtons down to several tens of nanonewtons — sometimes 
even below. 

A study of the available literature, applications, and a survey of the partners of 
a European research project’ resulted in the following specifications for force 
measurement in nanohandling: 


e Handling of micro-/nanomechanical parts: An example handling task 
mentioned several times is to grip a silicon part with a force-fit gripper. An 
Si cube with an edge length of 1 mm (density: 2.329 g/cm’, mass: 
2.329 mg) has a weight G of 22.85 uN. With a stiction friction coefficient 
of 0.1, the minimal required gripping force Fg according to Equation 6.9 is 
114 uN. A sensor with at least micronewton resolution is required to 
measure this force. To accomplish additional handling tasks, like assembly 
or joining, larger gripping forces are required, which are still in the range 
of millinewtons. The upper limit of the gripping force is defined by the 
tolerable surface pressure of the part’s material. For example, a force of 
1 mN causes a pressure of 0.4 MPa on an area of 50 x 50 um? , which is 
still safe taking into account the yield point of silicon (120 MPa) or nickel 
(30 MPa). However, the gripping area often cannot be used entirely 
because of very small structures on the part, such as edges from sawing the 
silicon; structures with membranes or cantilevers, efc., which can reduce 
the area significantly. 

e Micro-/nanomechanical characterization: The forces to be measured, 
e.g., during nanoindentation, range from 10 nN up to 500 mN and the 
required resolution is InN and better. 

e Handling of biological objects: The forces required to penetrate or deflect 
the membrane of a biological cell are in the micronewton range. A force of 
0.215 uN to 11 uN leads to a deflection of 5 um of a lipid double-layer 
[16]. These forces have to be measured with an adequate resolution in the 
nanonewton range. 


Calculation of (static) friction forces. The model usually applied to calculate 
static friction from normal force and friction coefficient is not necessarily valid in 
the micro- and nanoworld. In this domain, the friction coefficient is increasingly 
determined not only by the material of the interacting parts but also by their surface 
topography. However, the theory of friction at the macro- and nanoscale is not 
fully understood yet, and the macroscopic friction model has been applied above 
for a rough estimation of the gripping forces. 

Conclusions. From the specifications mentioned previously it is obvious that a 
compromise has to be found between a micro force sensor’s resolution and 
measurement range. The reproducibility of the sensor should be high enough to 
prevent damage during handling and assembly of micro/nano parts. However, for 
some applications, the measured values might not have to be calibrated exactly, 
and an estimation can be sufficient. It is not possible to develop a “universal” 





' EU FP5 Growth Project ROBOSEM (Development of a Smart Nanorobot for Sensor-based 
Handling in a Scanning Electron Microscope), 2002-2005. 
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micro force sensor which fulfills the wide spectrum of requirements. Ideally, 
several classes of force sensors for different force ranges will be developed, each 
with an appropriate resolution, measurement range, and size. 


6.2.5 Specific Requirements of Force Feedback for Microrobots 


When using microrobots for nanohandling, specific requirements have to be 
considered for the development of micro force sensors. These result, firstly, from 
the size and the motion principle of the robots (robot-specific requirements) and 
secondly from their application area and the corresponding environmental 
conditions (application specific requirements). 

Robot-specific requirements. The robots’ small size of only a few cubic 
centimeters, with correspondingly small grippers and end-effectors, requires the 
development of very small force sensors with at the same time high resolution. The 
integration of commercial force sensors is not possible due to performance 
limitations regarding size, measurement range, and resolution. In addition to the 
size, the microrobots’ mobility has to be taken into account. Some of the force 
measurement principles require a considerable amount of instrumentation. Force- 
sensitive structures have to be in the field of view of a microscope, or, in case of 
AFM-like systems, a detection unit for deflection measurement has to be moved 
relative to the robots’ motions. These requirements are difficult to meet in the case 
of mobile microrobots. Additionally, the stiffness of the cables required for 
actuators and sensors leads to a motion behavior which is difficult to model. Any 
additional cable should preferably be avoided. 

Application-specific requirements. A second complex of specific require- 
ments for the integration of micro force sensors into microrobots results from their 
application area. Especially in the case of scanning-electron-microscopy-based 
nanohandling applications, two main aspects have to be considered, namely the 
influences of the vacuum and the electron beam. 

Vacuum. A standard SEM requires a vacuum of 10 °Pa or better. Accordingly, 
for components being developed to operate inside the SEM, special guidelines 
have to be considered to avoid or at least to minimize a disturbance of the vacuum. 
This includes the choice of proper materials, design, manufacturing, and cleaning 
of vacuum-compatible components. The parts should not contain pores or cracks 
and should have a low gas pressure at room temperature. In particular, the 
outgassing of materials like glue has to be kept low. Additionally, hollow spaces 
must be avoided. The variation in heat conduction can be problematical for sensors 
and actuators used inside the SEM. In the vacuum, heat cannot be dissipated by 
convection, but only by conduction. Components can therefore easily heat up in the 
SEM. 

Electron beam. Components put into the SEM have to be compatible with the 
electron optics and the electron beam. Magnetic fields deflect the electron beam 
and lead to a distortion of the SEM image. The materials used should ideally be 
non-magnetic. Also, electrical fields caused by current-carrying wires or surfaces 
electrically charged by the electron beam can interfere with the SEM image. A 
suitable shielding or grounding has to be ensured so that the parts and their 
surfaces can discharge. If this is not done, local voltage concentrations can produce 
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fields which would deflect the electron beam and thus lead to image distortions. 
The electron beam can also interfere with components like force sensors. For 
instance, electrostatic forces caused by the electron beam can lead to problems 
during nanohandling due to charging of the parts, causing them to be attracted or 
pushed off or behave otherwise unpredictably. To solve these problems, the parts 
have either to be made of electrically conducting materials, or the non-conducting 
surfaces have to be coated with a thin gold layer. 


6.3 State-of-the-art 


In the 1930s, Ernst Ruska invented the electron microscope and created the basis 
for the development of the scanning electron microscope [17]. With this new 
technology, objects could be displayed with nanometer resolution for the first time. 
But it took another 50 years to develop additional tools and instruments that 
enabled research and manipulation of parts and materials at the nanometer scale. 
The invention of the scanning tunneling microscope (STM) and subsequently the 
atomic force microscope have especially to be mentioned; see Section 6.3.4 for 
more details. 

In parallel with STM-/AFM-based manipulation, a robot-based nanomanipu- 
lation approach emerged from combined research in robotics, precision mechanics, 
and microtechnology. The first research in this field was reported from Japan, 
where in 1990 an SEM-based nanomanipulation system was presented at the 
University of Tokyo [18]. 

It soon became necessary not only to transmit and scale human operations from 
the macroworld to the nanoworld by graphical user interfaces, telemanipulation 
devices, and dedicated actuators, but also to transmit information from the nano- to 
the macroworld. Ideally, information addressing all human senses would be 
acquired by sensors, transferred, scaled, and appropriately presented to the 
operator. However, so far, mainly vision and force sensors have been developed 
for this purpose. An overview of the state-of-the-art in force feedback for micro- 
and nanohandling will be given in the following sections. 


6.3.1 Micro Force Sensors 


Force sensing and force control has been an important topic in robotics research for 
more than 50 years [19, 20]. Various basic theories, force sensors, and control 
algorithms have been presented, implemented, and experimentally verified. 

In contrast, there are only a few micro force sensors available that can be used 
for robot-based micro- or even nanohandling. Existing force sensors can be 
differentiated according to the measurement principles, i.e., piezoresistive, 
piezoelectric, capacitive, or optical sensors. An overview of these sensors is given 
in Table 6.2. 


6.3.1.1 Piezoresistive Micro Force Sensors 
Several research groups have been working on the development of three- and six- 
axes force sensors for microhandling [21, 22], the manipulation of biological cells 
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[23], for robotics [24], and for micromechanical characterization [25]. All sensors 
have been manufactured using silicon micromachining and are based on integrated, 
piezoresistive elements, so-called piezoresistors. The sensitive element is either a 
crosswise structure or a so-called boss membrane. Details such as resolution, 
sensitivity or linearity are often not available from the literature. The lateral 
dimensions of the sensors range from 2.4 x 2.4 mm’ to 5 x 5 mm” and their 
maximum measurement range is between 0.01 N and 10 N. 


Table 6.2. Examples of micro force sensors 


















































Sensor Measurement Maximum Resolution Note Reference 
type principle force 

contact plezoresistive 1.6N 3 axes [21] 
force 

contact piezoresistive 1 mN 6 axes [22] 
force 

contact piezoresistive 5-10 N 3 axes [23] 
force 

contact piezoresistive 5N Fyin=0.1N [24] 
force 

contact piezoresistive 10-100 mN sub mN 3 axes [25] 
force 

gripping piezoelectric 2N [26] 
force 

contact capacitive 490-900 uN 0.01-0.24 uN 2 axes [27] 
force 

forcein optical 10 uN [28] 
general 

forcein optical uN range [29] 
general 

contact laser 500 uN 19 nN 1 axis [30] 
force 

contact optical +3nN,+3mN 1 axis [31] 
& 

gripping 

force 

contact optical 0.5 nN 1 axis [32] 
force 





The three-axes force sensor described in [25] is used for mechanical 
investigations of micromechanical structures (Figure 6.4a). The basic element is a 
boss membrane made out of silicon onto which a stylus has been glued. If a force 
acts on the pin, the membrane gets deformed and mechanical stress occurs in the 
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membrane. The areas of maximum stress are at the border of the membrane and at 
the intersection between membrane and boss (Figure 6.4b). Into these areas, 24 
piezoresistors have been implanted and connected to three Wheatstone bridges. 
Mechanical stress causes a change of the resistances and thus a change of the three 
bridge voltages. If the resistors are connected in a suitable way, the bridge voltages 
are proportional to the three components of the force acting on the pin. The dimen- 
sions of this sensor are 5 x 5 x 0.36 mm’, 


piezo- F 
resistors 
5 F 





a b 


Figure 6.4. Piezoresistive three-axes micro force sensor. a. Schematic view of the sensor 
(from [25], © Springer-Verlag 2001, with kind permission of Springer Science and Business 
Media); b. working principle (from [33], courtesy of Shaker Verlag). 


6.3.1.2 Piezoelectric Micro Force Sensors 

A piezoelectric micro force sensor for microhandling was described for the first 
time in [26]. The dimensions of the sensor were 16 x 2 x 1 mm’. A prototype was 
built but was neither characterized nor integrated into a microgripper. Information 
about the performance of this sensor, such as resolution or accuracy, is not 
available. 


6.3.1.3 Capacitive Micro Force Sensors 
A two-axes capacitive force sensor for the handling and characterization of 
biological cells was presented in [27], Figure 6.5a. 





Figure 6.5. Capacitive micro force sensor for biological cell handling. a. Schematic repre- 
sentation of the sensor; b. detail of the comb drives (from [27], © 2002 IOP Publishing, 
with kind permission of IOP Publishing). 
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The outer fixed frame and the inner moveable plate are connected by four 
springs. A force acting on the tip causes a movement of the inner plate and thus a 
change of the distance between each pair of the comb-like capacitors. The force 
can be calculated from the total capacity. The stiffness of the sensor, and thus its 
measurement range, can be adjusted by changing the dimensions of the springs. To 
be able to measure forces in the x- and y-directions, several comb drives were 
arranged perpendicularly (Figure 6.5b). Two capacitors in the middle of the inner 
plate were designed as reference capacitors for the signal-processing circuit. 


6.3.1.4 Optical Methods for Micro Force Measurement 

Strain and force measurement can also be realized using optical methods. For 
instance, either passive micro strain gages which are mechanically amplified can 
be used [28] (Figure 6.6), or strain in the nanometer range is measured using a 
light-optical microscope and image-processing tools [29]. By using this method, it 
is possible to measure forces with a resolution in the micronewton range. Both 
methods have been used for the characterization of micromechanical materials. 


vernier gauge 





tensile compressive 
strain strain 
<- o 


indicator beam 
slope beam 


test beam 
aN anchor 











anchor 


Figure 6.6. Optical force measurement using a passive, mechanically amplified micro strain 
gage 


A force sensor for a teleoperated micromanipulation system is described in 
[30]. The sensor is made of a cylindrical glass tube (length: 10 cm, diameter: 
500 um) with a tip of 20 um in diameter. The tube levitates between two plates 
made of diamagnetic material. Thus, mechanical friction can be avoided, and the 
resolution is improved. If a force acts on the glass tube, it is deflected and the 
deflection can be measured with a laser sensor. The resolution of 19 nN mentioned 
is only a theoretical value, which has been calculated from the stiffness of the 
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sensor setup (19 nN/um) and the resolution of the laser sensor (1 um). The entire 
setup has dimensions of 17 x 10 x 7 cm’. 
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Figure 6.7. Vision-based force measurement. a. Tracking of the contour of a deformed 
biological cell; b. distribution of forces on the cell membrane (courtesy of the Institute of 
Robotics and Intelligent Systems, ETH Zurich, http://www.iris.ethz.ch). 


A method of measuring the influence of a force acting on deformable, linear 
elastic objects is described in [31]. With a so-called template-matching algorithm, 
the degree of deformation of an object can be determined (Figure 6.7a). Using 
linear elastic theory, the force acting on the object can be calculated (Figure 6.7b). 
This method was validated using a silicon cantilever and a microgripper. With 
these test objects, a resolution of +3 nN and +3 mN, respectively, could be 
achieved. 
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Figure 6.8. Micromechanical force sensor for the characterization of biological cells 


In [32], a micromechanically manufactured force sensor for the characterization 
of living cells is presented. The sensor is made of a tip, which is fixed to frame via 
flexures (Figure 6.8). If the cells are indented with the tip, the corresponding force 
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can be determined by measuring the deflection of the flexures using a light-optical 
microscope. The force can then be calculated as a product of the sensor’s stiffness 
and the deflection. With a stiffness of a few nanometers per micrometer, the 
theoretical resolution of this method is 0.5 nN. 


6.3.1.5 Commercial Micro Force Sensors 

In spite of the importance of force feedback for nanohandling, only a few micro 
force sensors are commercially available. To some extent, this can be explained by 
the strong dependency between the development of microgrippers/end-effectors 
and micro force sensors. Due to integration issues, it is difficult — and to some 
extent impossible — to develop a force sensor without taking into account the 
design of the gripper/endeffector (Section 6.2.4). SensorOne Technologies 
Corporation, USA, is the only company offering a piezoresistive sensor based on 
silicon technology which can be used — after calibration — as a force sensor. The 
smallest commercially available multi-axes sensor is distributed by ATI Industrial 
Automation, USA. It is a six-axes force/torque sensor with a measurement range of 
up to 17 N/120 N-mm and a resolution of 0.78 mN/3.9 mN-mm. Due to its size, 
weight, and resolution, this sensor is unlikely to be useful for nanohandling. Its 
main application fields are robotics, micromanipulation [34], or dental research. 


6.3.2 Microgrippers with Integrated Micro Force Sensors 


Grippers and end-effectors are amongst the most important components of a 
handling and assembly system as they are in direct contact with the parts. The high 
number of publications dealing with the investigation of gripping principles 
demonstrates this importance: More than forty mechanical and micromechanical 
grippers have been developed worldwide within the last 15 years. Additionally, 
several suction and adhesive microgrippers, as well as contact-free handling me- 
thods, such as electrostatic grippers, have been reported. An overview of micro- 
grippers with integrated force sensors is given in Table 6.3. Additional micro- 
grippers, which have been integrated into robot-based micro- and nanohandling 
systems, are described in the Section 6.3.3. 

The first investigations on force measurement for microhandling were carried 
out in Japan [35]. To measure the gripping force and the position of the end- 
effectors simultaneously, four strain gages were integrated into a microgripper 
driven by piezoelectric bimorph actuators. Also, the miniaturized grippers des- 
cribed in [36-38] were developed with integrated strain gages to measure the 
gripping force and, in case of the last two publications, with additional strain gages 
to measure the deflection of the gripper jaws. 

A microgripper with an integrated force sensor based on a piezoresistive AFM 
cantilever was developed at the Fukuda Laboratory at Nagoya University, Japan 
[39]. In the center of the several-millimeters-long and 425 um wide end-effector, 
the thickness was reduced to 50 um. Within this area, four piezoresistive strain 
gages were implanted and connected into a full Wheatstone bridge. The achievable 
resolution of this force sensor is 0.1 uN. 
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Table 6.3. Overview of microgrippers with integrated force sensors 
































Sensor Measurement Maximum Resolution Note Reference 
type principle force 

gripping strain gages 15 mN [35] 

force 

gripping strain gages 130 mN +0.1 mN [36] 

force 

gripping strain gages several [37] 

force 10 mN 

gripping strain gages 500 mN [38] 

force 

gripping _ piezoresistive 2 mN 0.1 uN [39] 

force 

gripping piezoelectric qualitative [40] 

force measurement 

gripping capacitive qualitative Caesar’, not 
force measurement published 





Sensitive gripping of microobjects based on the piezoelectric effect was 
realized by [40]. The DC voltage required for opening and closing of the gripper 
via a piezoelectric actuator is superimposed with an AC voltage. This AC voltage 
has a smaller amplitude than the actuator signal and excites the end-effectors to 
oscillate at resonance frequency. If an object is gripped, the phase and amplitude of 
the sensor signal change and give an estimation of the relative change of force. 
One problem of this method is the visible oscillations, another one is the lack of an 
absolute force measurement. 

A gripper with force feedback based on quartz tuning forks was developed at 
the Caesar Research Center, Germany. The gripper is driven by two piezoelectric 
bimorph actuators. A tuning fork is fixed on each of the actuators; as end-effectors, 
glass needles were glued at the end of each tuning fork. The drawback of force 
sensors based on tuning forks is the lack of quantitative measurements: the sensor 
cannot be calibrated but provides only tactile feedback. Another drawback is that 
the frequency change used for force feedback can also occur due to a temperature 
shift or due to deposits on the end-effectors, e.g., from carbon contamination inside 
the SEM. Additionally, the vibrations of the tuning fork cause problems, since they 
are visible under a microscope and can hamper the handling of small parts. 


6.3.3 Robot-based Nanohandling Systems with Force Feedback 
Robot-based micro- and nanohandling is based on two different types of robots: 


(i) industrial robots (or: conventional robots) and (ii) microrobots, which can be 


? Caesar Research Center, Germany, http://www.caesar.de/ 
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either stationary or mobile. The latter have notable advantages regarding flexibility 
and modularity at the cost of difficult position sensing. 


6.3.3.1 Industrial Microhandling Robots 

Typical industrial robots used for microhandling are articulated robots, SCARA 
robots, and Cartesian robots. They have in common robust components like 
manipulators and grippers, a high stiffness, a high payload, and high positioning 
velocities. Their resolution and positioning accuracy is in the range of micro- 
meters. Compared to microrobots, sensors, and CCD cameras can be integrated 
quite easily. The use of this type of robot is determined by industrial applications 
such as the automated assembly of hybrid micro-electro-mechanical systems 
(MEMS) or microoptical systems in small to medium quantities. Although these 
robots are not suitable for nanohandling, some implementations of force feedback 
have the potential for miniaturization and use in microrobotics-/nanohandling and 
are thus given in the following (Table 6.4). 


Table 6.4. Overview of force feedback for industrial microhandling robots 





























Sensor Measurement Maximum force Resolution Note Reference 
type principle 

gripping fiber optical qualitative [41] 
force measurement 
gripping strain gages 538 mN accuracy: [42] 
force +14 mN 

gripping _ piezoresistive 6 mN [33] 
force 

contact fiber optical 40 mN/100 mN linearity: [43] 
& 7.1% / 20% 
gripping 

force 

forcein piezoresistive 200 uN AFM [44] 
general cantilever 

portable piezoresistive 10-3000 uN a few uN [45] 
gage 

portable piezoresistive uN range cantilever [46] 
gage type 





Fiber optical sensors for process control in microsystem assembly were 
developed by [41]. Two different solutions for integrated sensors were realized, 
and the usability of these sensors to measure gripping forces has been shown in 
principle (Figure 6.9). Using a grayscale picture, the presence of a gripped object 
and information about an increase or decrease of the gripping force could be 
gained. However, quantitative measurements were not possible. 
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Figure 6.9. Two different concepts of a fiber-optic-based gripping force sensor measuring 
intensity changes by using a. a small mirror; b. a gap which changes its size during gripping 
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Figure 6.10. Gripping force sensor based on strain gages. a. Actuator integrated into the 
gripper; b. strain gages glued on the actuators’ housings (from [42], courtesy of Shaker 
Verlag). 


Hence, a gripping force sensor based on strain gages was developed for the 
same gripper by [42], Figure 6.10a. To enable a simple and fast exchange of the 
grippers’ end-effectors without additional electrical connections, two strain gages 
have been glued on each of the two actuators’ housings, Figure 6.10b. If an object 
is gripped, the strain caused by the deformation of the actuators’ housing can be 
measured. Taking into account the change of the stiffness of this system, the 
gripping force can be calculated. This method has the drawback of low accuracy. 

Several different microgrippers for the assembly of microparts were developed 
by [33] (Figure 6.1la). The grippers were manufactured using plasma-enhanced 
dry etching of silicon and deep UV lithography of SU-8. The joints of the grippers 
were designed as flexure hinges; they are actuated by shape memory alloys. A 
double cantilever beam structure for stress concentration was integrated into the 
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gripper jaws, and piezoresistors were implanted in the areas with the highest 
mechanical stress (Figure 6.11b). The tensile and compression stresses during 
gripping cause a change of the resistance of the piezoresistors. A noticeable 
nonlinearity of the sensor’s output presumably results from the wires to the 
piezoresistors. As the wires are made of a sputtered gold layer, they deflect 
together with the flexures and thus change their resistivity analogously to the 
piezoresistors due to mechanical stress. 





a b 


Figure 6.11. Force feedback using piezoresistors. a. Microgripper with two integrated force 
sensors; b. double cantilever beam with stress concentrating structure (from [33], courtesy of 
Shaker Verlag). 


The goal of the work described in [43] was the development of grippers with 
integrated sensors for gripping and contact force measurement. The gripper is 
actuated by piezoelectric bimorphs. A parallel movement of the gripper jaws is 
guaranteed by specially designed flexures. Force-sensitive elements and optical 
fibers have been integrated into the end-effectors. If a contact force acts on the 
gripped object, it is being transmitted through static friction forces (between object 
and end-effector) to the contact-force-sensitive elements (Figure 6.12a). Gripping 
forces are transmitted in a similar way to the gripping-force-sensitive elements 
(Figure 6.12b). Both elements transform the forces into deflections, causing the 
free end of the optical fiber to move relative to the fixed one. Thus, the intensity of 
the light interfacing with the receiving fiber is modulated by the force, and the 
force can be determined by measuring the change in intensity. Contact forces can 
only be measured correctly when the gripper’s main axis is perpendicular to the 
working surface. 

Micromachined cantilevers with piezoresistors have been used at the Physi- 
kalisch-Technische Bundesanstalt, Germany, for two different applications: (i) as a 
sensor in a coordinate measurement machine [44], and (ii) as a portable gage, e.g., 
for the calibration of nanoindenters [45, 46]. For this purpose, both commercially 
available AFM cantilevers (Nanosensors GmbH, Germany; now: NanoWorld AG, 
Switzerland) and proprietary developments have been investigated and charac- 
terized. These investigations showed their principle usability as force sensors for 
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the measurement of forces up to approximately 3 mN. The development of the 
portable gage showed that micro forces can be measured with high sensitivity and 
good linearity using micromachined cantilevers with integrated piezoresistors. 
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Figure 6.12. Force sensors based on optical fibers. a. Contact force sensor; b. gripping force 
sensor. 


6.3.3.2 Microrobots Outside the Scanning Electron Microscope 

Microrobots are used for various applications from life science, microelectronics, 
materials research and, MEMS. Either commercially available, complete solutions 
are applied — e.g., Kleindiek Nanotechnik, Klocke Nanotechnik, Zyvex, Sutter 
Instruments, Eppendorf, etc. — or custom-made solutions are developed, which are 
often based on commercial nanopositioning stages from, e.g., Physik Instrumente, 
SmarAct, attocube, Nanomotion, efc. For these systems, a few microgrippers are 
available on the market. However, gripping or contact force measurement has not 
been integrated into any of them. 

By contrast, there are several research publications describing the development 
of force sensors for micro- and nanohandling (Table 6.5). Three different types of 
force sensing have been implemented: (1) discrete strain gages, (ii) integrated 
piezoresistive strain gages, and (iii) piezoelectric force sensors based on PVDF 
films. 

Discrete strain gages. For the measurement of gripping and contact forces in 
microrobotics, mainly discrete strain gages have been used so far. These are 
commercially available metal-foil or semiconductor strain gages which can directly 
be glued onto microgrippers and end-effectors. 

Three publications describe the development of microhandling systems work- 
ing under a light microscope [47-49]. Each of these systems uses piezoelectrically 
actuated microgrippers with integrated gripping force feedback based on semi- 
conductor strain gages. The ideal positions of the strain gages, i.e., the areas with 
the highest mechanical stress, have been determined using finite-element methods. 
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The measurement range is within millinewtons. Information about the achievable 
resolution is not available. 

A different approach for microhandling was pursued at the University of Cali- 
fornia [53]. The “gripper” is made of two probes, each having one degree of 
freedom. These probes are mounted perpendicularly to each other. Three semi- 
conductor strain gages have been glued onto them, two for measuring the position 
and one for measuring the force. With this setup, it is possible to grip parts smaller 
than 1 mm and rotate them around their vertical axis. At the same time, the 
gripping force can be measured with sub-millinewton resolution. 


Table 6.5. Force feedback for microrobots operating outside the SEM 



































Sensor Measurement Maximum Resolution Note Reference 
type principle force 

gripping strain gage 22 mN [47] 
force 

gripping strain gage 23 mN [48] 
force 

gripping strain gage 69 mN [49] 
force 

gripping _— piezoresistive 500 uN 0.2 uN [50] 
force 

gripping _— piezoresistive 600 uN 2 nN [51] 
force 

gripping piezoelectric 6 mN several [52] 
force 10 uN 

gripping strain gage >4mN < 0.4 mN [53] 
force 

contact piezoelectric uN range [2] 
force 





Integrated strain gages. The smallest commercially available discrete strain 
gages have lateral dimensions around 1 mm. With that size, they are often already 
too large for integration into ever smaller grippers and end-effectors. Thus, piezo- 
resistive strain gages are more and more directly integrated into areas with high 
mechanical stress by ion implantation. This work was pioneered by M. Tortonese, 
who developed the first piezoresistive AFM cantilever at IBM Almaden Research 
Center [54]. 

One of the first examples of the transfer of this technology to microrobotics is 
the tactile, silicon-based microgripper developed at ETH Zurich [50] (Figure 
6.13a). The gripper is made of two fingers — a bimorph actuator and a piezo- 
resistive force sensor — which both serve as end-effectors at the same time (Figure 
6.13b). The length of the fingers is 1.5 mm, their width between 80 um and 
240 um, and their thickness is 12 um. Objects with a size up to 400 um can be 
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gripped. Two different types of this gripper were developed. One version had four 
piezoresistors connected into a full Wheatstone bridge and one smaller version had 
only two piezoresistors. An analog PI controller was implemented to allow for 
force-controlled gripping of microparts. 
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Figure 6.13. Tactile microgripper. a. Comparision to a match (from [50], courtesy of 
Georg Greitmann); b. cross.section of sensor and actuator finger 


lower finger 


upper finger Modüle 


module 








V/ : ; 
a . piezo-electric 





terminal 4 force sensor er evice 
base poe 
a b 


Figure 6.14. Piezoresistive micro force sensor. a. Detail with glass needle as endeffector; 
b. integrated into a two-fingered robot hand (from [51], © 2001 IEEE). 


A similar force sensor was developed at the Mechanical Engineering 
Laboratory, Japan, in co-operation with Olympus [51]. By using semiconductor 
technologies and micromachining, two strain gages were integrated into a silicon 
cantilever (Figure 6.14a). Additionally, the cantilever was etched down to a thin 
membrane to amplify the mechanical stress. The strain gages have an opposite 
gage factor to increase the resolution of the sensor. Due to the lack of a suitable 
calibration, the resolution could only be calculated theoretically to be 2 nN. A thin 
glass needle was glued to the end of the force sensor as an end-effector. Two 
sensor/end-effector devices were integrated into a two-fingered micromanipulator 
(Figure 6.14b). By using two sensors and mounting them rotated by 90°, a force 
feedback in two axes could be realized. 
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Piezoelectric micro force sensors. The use of piezoelectric force sensors based 
on films of polyvinylidene fluoride (PVDF) is a new approach in microrobotics 
research. PVDF is a semi-crystalline polymer consisting of monomer chains of 
(-CH2-CF,-),. It has strong piezoelectric properties due to a large electro- 
negativity of the fluorine atoms in comparison to the carbon atoms. PVDF creates 
electrical charges like other piezoelectric materials only when the film is subjected 
to mechanical, acoustic, or thermal stress. 

At the Korea Institute for Science and Technology (KAIST), a force sensor 
based on PVDF has been integrated into a microgripper [52]. A commercially 
available film with nickel electrodes on both sides has been used (Measurement 
Specialties Inc., USA). The film was cut into 2 x 6 mm?’ slices and coated with a 
thin parylene layer for protection, as well as to enhance the mechanical stiffness. 
The resulting thickness of the sensor was 36 um. The resolution was specified to 
be several tens of micronewtons (the load cell used for calibration had a resolution 
of 50 uN). 





Figure 6.15. Schematic drawing of a two-axes PVDF micro force sensor 


A commercial five-axes IC wafer probe was used as a micromanipulation 
system at Michigan State University, USA, to automate the assembly of 
micromirrors. The mirrors usually lie on the substrate’s surface after production 
and have to be brought into an upright position where they are locked 
mechanically. A two-axes PVDF-based force sensor was developed to measure the 
forces during the lift process [2]. For each axis of the sensors, two PVDF films are 
connected in parallel, and the two axes are assembled in series and rotated by 90° 
(Figure 6.15). The entire sensor has a length of more than 4 cm and a height of 
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more than 1 cm. It was calibrated by measuring its deflection under a light 
microscope; the stiffness required to calculate the force was determined from the 
Young’s modulus and the geometry of the sensor. 


6.3.3.3 Microrobots Inside the Scanning Electron Microscope 

More and more applications in micro- and especially nanotechnology hit the light- 
optical microscope’s resolution limit of about 400 nm. Smaller objects, such as 
carbon nanotubes or nanowires with dimensions of several hundreds of nanometers 
down to only a few nanometers, cannot be resolved individually with a light- 
optical microscope. The scanning electron microscope has been used for several 
decades now as a flexible microscope and analysis instrument. It combines a high 
resolution — down to about | nm for field-emission instruments — with a high depth 
of field. As a result, handling and characterization of parts is increasingly 
performed inside the SEM. Table 6.6 gives an overview of nanohandling systems 
with force feedback working in an SEM. 


Table 6.6. Overview of microrobots inside the SEM with force feedback 














Sensor Measurement Maximum Resolution Note Reference 
type principle force 

contact strain gage 0.5-3.5 N 130-180 uN 3 axes [55] 

force 

contact strain gage 20 mN 14 uN 1 axis [56] 

force 

contact strain gage 40 mN 309-391 uN 3 axes [57] 

force 





The first research activities within this field were carried out at the University 
of Tokyo [18], where, for a duration of more than ten years, the so-called Nano 
Manufacturing World was developed. A microassembly system consisting of a 
miniaturized robot arm and a positioning stage with a total of twelve translatory 
and rotatory axes for coarse and fine positioning was installed in a stereo SEM. 
Forces occurring during assembly are measured with a three-axes strain gage 
contact force sensor based on parallel plates. The forces are transformed into 
acoustic signals; additionally, a haptic device provides force feedback for the 
operator [55]. 

A similar nanohandling system for automated handling of microobjects based 
on visual and force feedback is described in [56]. Additionally to the processing of 
the images from the SEM and a light-optical microscope, a force sensor has been 
installed inside the SEM. This sensor measures the normal force acting on the work 
plate. It is made of a cantilever (dimensions: 10 x 8 x 0.05 mm’) onto which four 
strain gages have been glued. The forces during assembly have values of about 
1 mN and can be sufficiently resolved with this sensor. 

At Tokai University another microhandling system has been installed inside an 
SEM in cooperation with Hitachi [57]. The system is made up of two identical 
micromanipulators each with three Cartesian axes for coarse and fine positioning 
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(Figure 6.16a). The sample is mounted on a stage with three Cartesian and one 
rotary axes. A three-axes contact force sensor was mounted on each of the micro- 
manipulators (Figure 6.16b). The sensors are based on semiconductor and metal- 
foil strain gages, respectively. With these sensors, the contact forces between 
manipulator and environment or the manipulated part can be measured. The sensor 
information is transmitted to the operator via a haptic user interface. 





Figure 6.16. Microhandling system at Tokai University. a. Positioning stages integrated 
into an SEM; b. three-axes strain-gage-based contact force sensor (from [57], © 2003 
IEEE). 


Other microrobot-based handling systems inside the SEM have been 
developed, especially for the characterization of nanoobjects like carbon nanotubes 
[58-60]. Furthermore, different nanopositioning and nanohandling systems have 
been commercially available for several years. All these systems have in common 
piezoelectric actuators which can be controlled via teleoperation. Some have 
integrated position sensors; however, force feedback is not integrated into any of 
these devices. 


6.3.3.4 Mobile Microrobots 

For several years now, mobile microrobots have been an important topic of world- 
wide research activities. Their advantages compared to stationary microrobots 
described above are a greater flexibility, a larger working space and the possibility 
of using several cooperating robots to accomplish a task. Mobile microrobots have 
been developed, for instance, within European research projects (MiCRoN’, 
I-Swarm’) as well as at the NanoRobotics Laboratory of the Ecole Polytechnique 
de Montréal, Canada [61]. These projects focused on the development of 
miniaturized, autonomous robots with integrated electronics and wireless com- 
munication as well as the control of cooperating robots and robot swarms. The 





3 EU FP5 IST Project MiCRoN (Miniature co-operative robots advancing towards the nano- 
range), 2002-2005. 

4 BU FP5 IST FET-open Project I-Swarm (Intelligent Small World Autonomous Robots for 
Micro-manipulation), 2004—2008. 
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microrobots were not designed to work inside a scanning electron microscope. 
Micro force sensors have not been developed or integrated within these projects. 

Mobile microrobots working inside an SEM have been developed within two 
European research projects (MINIMAN°, ROBOSEM®) and at the Aoyama 
Laboratory in Tokyo [62]. Within MINIMAN, a prototype of a force-sensing 
system based on strain gages was realized and its repeatability, precision, and 
linearity quantified [63]. A simple method that links system parameters and system 
performance was presented and verified concerning cantilever-type end-effectors 
for microgrippers. The resolution obtained in an overall measurement range of 
10 mN was 150 uN. 





a b 


Figure 6.17. Piezoresistive cantilever-type micro force sensor. a. Top and bottom view of 
the sensor with piezoresistors, structured gripping area and cavity for mounting; b. sensor 
integrated into a microgripper, which is mounted on a micromanipulator (reprinted from 
[64], © 2004, with permission from Elsevier). 


Within ROBOSEM, a set of sensors for the measurement of gripping forces in 
the range of 10°’ N to 10' N has been developed [64]. The force sensor consists of 
a silicon cantilever beam with force-sensing elements located at its support (Figure 
6.17a). The piezoresistive circuit contains four piezoresistors in a full Wheatstone 
bridge setup with additional compensation of temperature and offset voltage. The 
cantilever beam was used as an end-effector of the gripping device. Sensors with 
different cantilever lengths ranging from 1 mm to 3 mm and thicknesses of up to 
100m were developed to cover a wide range of gripping forces. These sensors can 
easily be integrated into different types of microgrippers and end-effectors (Figure 
6.17b). Tools equipped with these force sensors can be used for handling micro- 
objects under ambient and vacuum conditions. 





$ EU FP4 Esprit Project MINIMAN (Miniaturised Robot for Micro Manipulation), 1998- 
2001. 

€ EU FP5 Growth Project ROBOSEM (Development of a Smart Nanorobot for Sensor-based 
Handling in a Scanning Electron Microscope), 2001-2005. 
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6.3.4 AFM-based Nanohandling Systems 


The manipulation of nanoscale objects and of matter at even the atomic level is a 
comparatively new research field. It has been opened up by the invention of the 
scanning tunneling microscope by Binning and Rohrer in the early eighties [65] 
and the subsequent invention of the atomic force microscope by Binning et al. in 
1986 [66]. Even though both types of microscopes were designed for investigating 
surfaces of electrically conductive materials and insulators, respectively, it was 
soon discovered that they could also be used to modify these surfaces and 
manipulate objects as small as atoms (Chapter 1). Eigler and Schweizer were the 
first to manipulate atoms in an STM on a single-crystal nickel surface when they 
formed the IBM logo out of 35 xenon atoms [67]. 

Force measurement with nanonewton resolution and below is inherently integ- 
rated into AFMs. Although they were originally only intended for imaging, they 
are nowadays also used for nanohandling with force feedback. As described in the 
following sections, the design of standard AFMs limits their usability as nano- 
handling systems, but recent developments are likely to overcome these limitations 
and convert AFMs into flexible instruments for imaging and manipulation at the 
nanoscale. 


6.3.4.1 Commercial and Custom-made AF Ms for Nanohandling 

Within the last few years, several nanohandling systems either based on 
commercial AFMs or on custom-made AFM instruments, have been developed 
[68-73]. At the Korea Institute of Science and Technology, an AFM has been built 
especially for nanohandling [74]. It is made up of a nanomanipulator with three 
degrees of freedom and piezoresistive AFM cantilevers. 
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Figure 6.18. Force-controlled pushing using an AFM cantilever. a. Side view of the 
experimental setup with AFM cantilever, workpiece, and reference object; b. top view of 
the configuration at the start (left) and end of the alignment process (right) (from [75], 
courtesy of SPIE). 


Zesch et al. used a piezoresistive AFM cantilever for force-controlled pushing 
of sub-millimeter-sized silicon parts on a silicon substrate [75]. The substrate can 
be positioned using a precision table. The cantilever is mounted on a piezoelectric 
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positioning stage (Figure 6.18a). With this setup, microobjects can be aligned 
laterally relative to each other (Figure 6.18b). By measuring the contact forces 
between object and AFM cantilever it is possible to draw conclusions as to whether 
the object is moved, if it got lost, or if it hit an obstacle. The measured forces are 
typically between of 0.8 uN and 3.5 uN during pushing and twice as high when 
static friction has to be overcome. The biggest problems of this method are 
instabilities during pushing, as well as non-predictable movements or jumping of 
the microobject. 

The basic element of the nanohandling system developed at the University of 
Minnesota, USA, is an IC probing station into which a three-axes nanopositioning 
unit was integrated [76, 77]. The probing station was combined with an AFM head, 
which measures the deflection of an AFM cantilever using a laser beam. This sys- 
tem was mainly used to develop combined force/vision control and to investigate 
adhesion forces in nanohandling. 


6.3.4.2 AFMs Combined with Haptic Devices and Virtual Reality 

As identified in Chapter 1, the main drawback of standard AFMs is the lack of 
visual feedback of the manipulation process in real-time: each operation has to be 
designed offline based on a static AFM image and transferred to the AFM to carry 
out the task in open-loop. The success of the operation has subsequently to be 
verified by a new image scan. Obviously, the cycle of scan—design—manipulate— 
scan is very time consuming, because it usually takes several minutes to obtain an 
AFM image. 

To overcome this problem, AFMs combined with haptic devices (e.g., 
NanoMan from Veeco, USA; NanoManipulator™ from 3rdTech, USA; 
NanoFeel™300 manipulator from NanoFeel, Switzerland; Omega Haptic Device 
from Force Dimension and Nanonis, Switzerland), and virtual reality (VR) 
interfaces to facilitate feedback during nanomanipulation, have been developed, 
e.g., [78, 79]. Haptic devices provide the operator with real-time force feedback 
but they cannot compensate for visual feedback. Virtual reality interfaces can 
display a static, virtual environment, and a dynamic position of the tip. However, 
they do not reflect any environmental changes in real-time, and the operator is still 
“blind”, so to speak. Compared to standard AFMs, both techniques represent a 
significant improvement for the user, but they still do not eliminate the need for an 
SEM to visualize manipulation and processing in the nanoworld in real-time. 


6.3.4.3 AFMs Integrated into Scanning Electron Microscopes 

As mentioned in Chapter 1, AFMs integrated into SEMs combine the advantages 
of both tools into one instrument, e.g., allowing the combination of nanoscale 
chemistry, crystallography imaging via electron-matter interactions with 
information from tip-sample interactions like topography, or magnetic/electrostatic 
force imaging. Several of such hybrid AFM/SEM systems have been developed in 
the research [80-85]. Three commercial AFMs for integration into an SEM are or 
were available on the market. The Observer from Topometrix (now Veeco 
Instruments, USA) has been reported on twice in the literature [86, 87], but is not 
sold anymore. Nanonics Imaging Ltd., Israel, commercializes a modified version 
of their AFM for use inside an SEM. The third vendor is Kleindiek Nanotechnik 
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GmbH, Germany, who offer an AFM for SEMs based on one of their 
micromanipulators. 

State-of-the-art is the manual positioning of an AFM tip with conventional 
atomic force microscopes integrated into large-chamber SEMs for topography 
imaging as well as crystallography, chemistry, and surface morphology analysis. 
Only Williams ef al. [86] have reported on the use of the Topometrix Observer for 
the controlled placement of an individual carbon nanotube on a MEMS structure in 
an SEM. None of the above-mentioned systems was designed for manipulation or 
assembly. 


6.4 Conclusions 


The measurement of micro- and nanonewton forces is one of the central challenges 
in nanohandling. This is shown by the large number of publications and amount of 
research in that field. However, from the state-of-the-art it can be seen that no 
micro force sensor exists so far — neither in research nor on the market — that can 
fulfill the many requirements of microrobotics. Often, the sensors are just too large 
to be integrated into microgrippers or end-effectors. Mostly, their performance in 
terms of resolution or accuracy is poor. Force feedback solutions, which have been 
developed for industrial robotics, do not aim at nanohandling applications and are 
not suitable due to performance limits. 

One of today’s most promising solutions for force measurement in micro- 
robotics is the use of piezoresistive sensors. They can be manufactured with the 
same semiconductor processes as microgrippers and thus be directly integrated into 
the grippers, requiring no additional assembly. Another possibility is to implant 
piezoresistors into cantilever-like end-effectors, which can then be used as micro 
force sensors and, at the same time, modular and easily exchangeable gripper jaws. 
Also, the results of force sensing based on capacitive sensors and image processing 
(Chapter 4) indicate the potential of these techniques for future developments in 
force feedback at the nanoscale. 

AFMs can be used as nanohandling instruments with an already integrated, 
high-resolution force feedback. Combined with novel tools, such as dedicated 
micro-/nanogrippers, and integrated into scanning electron microscopes providing 
almost real-time vision feedback, AFMs promise to become a key instrument for 
the reliable handling and characterization of nanoscale objects. 
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7.1 Introduction 


The progressive reduction of integrated circuits is reaching the limit of structures 
that are realizable by lithographic methods. Intel is announcing 45 nm transistors 
for the year 2007. Using electron beam lithography, structures down to 10 nm are 
producible. But this will be the absolute lower limit for lithographic structuring, so 
that new materials have to be developed for a further miniaturization of structures. 

Since their discovery in 1991 by [1] of NEC laboratory in Japan, carbon 
nanotubes (CNTs) have turned out to be the most promising material for computer 
science and nanotechnology. Due to their remarkable structure and properties, 
CNTs allow for new and unique applications. After 15 years of intensive research 
and development, the commercialization has already begun in some domains. 
Application areas with the highest patent applications are field emitters, 
nanoelectronics, polymer composites, and new electrodes for batteries and fuel 
cells. 

Great progress has also been achieved in characterizing carbon nanotubes [2]. 
In addition to different methods such as atomic force microscopy (AFM), scanning 
tunneling microscopy (STM), transmission electron microscopy (TEM), and scan- 
ning electron microscopy (SEM), many spectroscopic and diffractional methods 
are used for a comprehensive characterization of CNTs. For the meantime, first 
characterization devices based on fluorescence and absorption spectra are commer- 
cially available. However, a systematic and reproducible analysis of the mecha- 
nical and electrical properties of individual CNTs is still missing in many cases. 

This chapter focuses on SEM-based nanomanipulation and the character- 
rization of individual carbon nanotubes by a microrobot-based nanohandling 
station. Mechanical properties such as the Young’s modulus, as well as electrical 
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properties such as the conductivity, are analyzed. Therefore, different end-effectors 
like AFM cantilevers and electrothermal nanogrippers are used and combined with 
real-time SEM image processing, resulting in a flexible and multifunctional 
nanohandling station. At the same time, automation of particular nanohandling 
steps requires the development of suitable control system architecture. 


7.2 Basics of Carbon Nanotubes 
7.2.1 Structure and Architecture 
Carbon nanotubes consist of a two-dimensional sheet of carbon atoms coiled up to 


a cylinder. The carbon atoms are hexagonally arranged and o-bonded. CNTs ap- 
pear in one of two configurations (Figure 7.1). 





Figure 7.1. Schematic models for CNTs: a. SWCNTs; b. MWCNTs 


Firstly, there are single-wall carbon nanotubes (SWCNTs) that are made up 
of one single carbon layer (Figure 7.1a). The typical diameter of SWCNTs is 
between 0.4 and 3nm. SWCNT ropes can also be generated through a self- 
organizing process. Here, van der Waals forces keep the individual SWCNTs 
together, forming a triangular lattice with a lattice constant of 0.34 nm. Secondly, 
there are multiwall carbon nanotubes (MWCNTs) containing several coaxial 
carbon layers with a typical diameter of 2 up to 100 nm (Figure 7.1b). The length 
varies from several nanometers up to several millimeters for both configurations. 

An SWCNT is uniquely characterized by its chiral vector (Figure 7.2): 


C=na,+ma,, (7.1) 


where a, and a, are the two unit vectors of the carbon sheet and (n, m) a set of 
two integers. SWCNTs are built by rolling up the carbon lattice such that the 
starting point A and the ending point B of the chiral vector C are superimposed. 
The chiral angle @ is defined with respect to the unit vector a, and runs from 0° to 
30°. The lattice vector T is shown as well. The diameter D of an SWCNT is given 
by: 
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D= čl = avn +m +nm ’ (7.2) 
mT mT 





where a is the length of the unit vector and amounts to 0.249 nm. 
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Figure 7.2. Definition of the chiral vector within a carbon lattice 


SWCNTs with (n, 0) are called zigzag nanotubes, while SWCNTs with (n, n) 
are armchair nanotubes. All other tubes with arbitrary pairs of integers (n, m) are 
referred to as chiral nanotubes. Figure 7.3 shows the three different types of 
SWCNTs. 





Figure 7.3. The three different types of SWCNTs: a. zigzag; b. armchair; c. chiral 
7.2.2 Electronic Properties 


The electronic properties of carbon nanotubes are a matter of particular interest due 
to their high application potential in nanoscale electronic devices. The ideal 
symmetric structuce of carbon nanotubes is leading to unique electronic properties 
like quantum effects and ballistic electron transport. 
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SWCNTs can have metallic or semiconducting characteristics, depending on 
their chirality. Theoretical calculations have shown that SWCNTs are metallic 
conducting, if: 


(n—m) = 3k, (7.3) 


where k=0,1,2,.... 

This means that one third of all possible SWCNTs are metallic and two thirds 
are semiconducting. For example, all armchair SWCNTs show metallic conduc- 
tivity. The exact band gap of semiconducting SWCNTs depends on their diameter 
and can be calculated by: 


2a 
Egne, (14) 
where a, is the nearest neighbor distance and amounts to 0.142 nm and 7, is the 
nearest neighbor transfer energy with y, = 2.7 eV [3]. 

The typical band gap for SWCNTs is below 1 eV. SWCNTs show ballistic 
electron transport without any electron scattering over the whole CNT length. So 
the ideal resistivity of an SWCNT is independent of the length. In reality, however, 
the ballistic transport of electrons and thus the conductivity of SWCNTs depends 
on the number of defects in the CNT. Defects can act as scattering centers and thus 
decrease the mean free path L,, of an electron, given by: 


Ln =V pt, 


m m? (7.5) 
where v, is the Fermi velocity and r,, the characteristic relaxation time. 

For MWCNTs, the determination of the conductivity is more complex, since it 
consists of multiple walls and thus depends on the chirality of every individual 
shell. An MWCNT can be considered as a bundle of parallel conductors, each 
having a different band structure. First experimental measurements have produced 
different results. 

[4] identified that an MWCNT at low bias shows ohmic behavior and the 
electrical conductivity decreases analogously to the number of outer shells being 
removed from the MWCNT by current-induced oxidation. This effect of multilayer 
conductivity can be explained by shell-to-shell interaction. The coupling between 
the inner and outer shells is shown to be completely frozen out for temperatures 
below 90 K. Typical MWCNT resistivity ranged between 5 and 15 KQO. [5] pre- 
sented the fact that the conductivity of MWCNTs is independent of length, and 
that the electron transport seems to be quasi-ballistic, similar to SWCNTs. 

Recapitulating, the conductivity of an MWCNT depends on three parameters: 
the contact resistance between CNT and electrodes; the resistivity of individual 
shells; the shell-to-shell resistivity. 
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7.2.3 Mechanical Properties 


Due to the strong covalent o-bonding between the neighboring carbon atoms 
forming a nanotube, the tensile strength and Young’s modulus of CNTs exceed 
those of steel by magnitudes, which makes CNTs excellent for reinforced com- 
posite materials. 

Theoretical and experimental results have attested a Young’s modulus of about 
1 TPa and a tensile strength of about 150 GPa [6-11]. On the one hand, the 
Young’s modulus is reported to be independent of the tube diameter, tube chirality, 
and number of tube layers by using an empirical force-constant model [7]. But on 
the other hand, newer publications show effects of diameter, chirality, and number 
of layers by means of a molecular structural mechanics approach [9]. Thus, for a 
thorough understanding of the elastic properties of carbon nanotubes, systematic 
experimental measurements have to be carried out. 

MWCNTs can be seen as nested SWCNTs. The interactions between adjacent 
layers are mainly due to van der Waals forces. Compared to the covalent bonds 
between the carbon atoms within the graphene sheet, the van der Waals interaction 
is rather weak. As a result, relative slippage between the layers in an MWCNT can 
occur, and MWCNTSs can act as low-friction nanoscale linear bearings [12]. The 
structural flexibility was investigated by means of observations and atomistic 
simulations of the bending of SWCNTs and MWCNTs under mechanical stress 
[13]. The results show that single and multiple kinks can be observed, dependent 
on the bending curvature. Despite the occurrence of kinks, CNT bending is fully 
reversible and elastic up to very large bending angles of about 110°. 

Regarding a cantilevered, free-standing MWCNT that can be assumed as a 
cylinder with given diameter (Figure 7.4), one can obtain a relation between the 
deflection 6 of the MWCNT and its Young’s modulus £. Using a standard beam 
deflection formula according to [14], the Young’s modulus F is given by: 


F(6)P 
316 





> (7.6) 


where / is the length of the CNT, 6 the deflection of the CNT, F(ô) the appendant 
force acting on the CNT, and Z is the geometrical moment of inertia of a solid 
cylinder given by: 


I =nd‘ /64, (7.7) 


where d is the diameter of the CNT. 
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Figure 7.4. Deflection of a cantilevered CNT 
7.2.4 Fabrication Techniques 


Since the discovery of CNTs, when they were fabricated by arc discharge, a lot of 
additional techniques have been developed for the production of carbon nanotubes. 
The most common production techniques today are are discharge [15, 16], laser 
ablation [17, 18], and chemical vapor deposition [19, 20]. 


7.2.4.1 Production by Arc Discharge 

Arc discharge is a relatively simple and low-priced production technique for high- 
quality CNTs. For CNT production by arc discharge (Figure 7.5), two graphene 
electrodes are arranged in a vacuum chamber filled with helium. By impressing a 
voltage on the electrodes and decreasing the displacement between cathode and 
anode, an arc discharge occurs with a current flow of about 100 A. Due to this arc 
discharge, the carbon of the electrodes is evaporated and precipitates in the form of 
CNTs on the cathode. The yield obtained from arc discharge is up to 30%. Both 
SWCNTs and MWCNTs with only partial defects are created. By adding a metallic 
catalyst, the growths of SWCNTs can be enhanced. The main disadvantages of this 
method are the short length of nanotubes and their random size and alignment. The 
huge portion of byproducts require a following complex cleaning process. 
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Figure 7.5. Production of CNTs by arc discharge 
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7.2.4.2 Production by Laser Ablation 

Another method for producing CNTs called laser ablation is shown in Figure 7.6. 
For this method, a laser beam is directed at a graphite target, vaporizing parts of it. 
The vaporized carbon particles are carried by an argon gas flow within the vacuum 
chamber towards a copper collect vessel, where the carbon nanotubes grow. The 
yield of this method is up to 70% by weight. Mainly defect-free SWCNTs or ropes 
of SWCNTs are created. By varying the process temperature, the diameter of the 
CNTs can be affected. The disadvantages are the high costs due to the necessity of 
having a high-powered laser and, as in the case of arc discharge, the lack of control 
of the deposition area and the huge portion of byproducts. 

Figure 7.7 shows an SEM micrograph of typical MWCNTs produced by arc 
discharge and laser ablation. The CNTs are available in the form of powder 
forming a lump of CNTs. The separation of individual CNTs is, therefore, hard to 
achieve. 
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Figure 7.7. MWCNTs produced by arc discharge and laser ablation with diameters ranging 
from 60 nm up to 100 nm and length from 0.5 um up to 600 um 


7.2.4.3 Production by Chemical Vapor Deposition (CVD) 
Chemical vapor deposition (Figure 7.8) uses a chemical reaction, which 
transforms gaseous molecules, called precursors, into a solid material. 
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Figure 7.8. Production of CNTs by chemical vapor deposition 


For producing CNTs, small nickel particles with a diameter of about 30 nm are 
brought onto a silicon substrate (wafer). A carbon-containing gas, such as methane, 
flows over the wafer surface. At temperatures of about 600°C, the gas disintegrates 
and the released carbon atoms form tubes using the nickel particles as condensation 
seeds. The yield can be up to 100% by weight. Length, alignment, and position of 
the CNTs can be affected by varying the process parameters. In this way, long and 
perfectly aligned CNTs can be achieved (Figure 7.9). Another advantage is the 
purity of the CNTs, as the CVD method produces pure CNTs without any 
byproducts. However, due to the low process temperature, the CNTs often exhibit 
defects. 
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Figure 7.9. Array of MWCNTSs produced by chemical vapor deposition. CNT diameter is 
about 500 nm, length 6 um, and pitch 10 um. 


7.2.5 Applications 


A multitude of possible applications for CNTs [21] arise from their remarkable 
physical properties, such as their high Young’s modulus, high electrical con- 
ductivity, and high aspect ratio. For example the tensile strength of CNTs is in the 
order of 10° Pa, which corresponds to the multiple tensile strength of high-modulus 
steel. CNTs can carry a current density of about 10° A/cm’, which is 1000 times as 
much as that of copper. In the following sections, the four most realistic appli- 
cations, besides the development of CNT-based sensors [22], are presented. 
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7.2.5.1 Composites 

The high Young’s modulus of about 1 TPa and the high electrical conductivity 
make CNTs excellent for reinforced and conducting composites. For example, 
such conducting composites can be used for electrostatically applying paint onto 
car components or can act as antistatic shieldings. Another application is 
transparent electronic conductors in the display industry for the development of 
flexible displays on plastic substrates. 

Due to their mechanical properties, CNTs have attracted much interest for 
light-weight structural composites and are likely to exceed the performance of 
carbon-fiber, reinforced materials. Today, such CNT-reinforced composites are 
already being applied in the serial production of tennis rackets, in order to improve 
their rigidity and damping features. 


7.2.5.2 Field Emission 

The high aspect ratio and the huge current density make CNTs ideal field emitters 
under an intense electric field. CNTs are, therefore, the perfect electron source to 
build new high-resolution and very flat field-emissive displays (FEDs). The 
principle is similar to CRT (cathode ray tube) displays. Electrons are accelerated 
towards a phosphor layer, creating a light spot (pixel). But in contrast to old CRT 
displays in a FED, every pixel has its own electron source — a multitude of 
vertically aligned CNTs. Figure 7.10 shows the schematic of a FED. 
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Figure 7.10. Schematic of a field emissive display 





Between the cathode and gate-electrode a discharge voltage is applied, so that 
the CNTs start to emit electrons. By applying an acceleration voltage between the 
gate-electrode and anode, these electrons are directed towards the subpixel. Three 
subpixels form a pixel. The advantages of FEDs over LCDs are higher power effi- 
ciency and a wider operating temperature range. 

Another application for field-emitting CNTs are electron guns for scanning 
electron microscopes (SEMs) and transmission electron microscopes (TEMs). The 
small energy spread of the electrons and the high current density will improve the 
lateral resolution and the brightness of the next generation of microscopes. 
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7.2.5.3 Electronics 

Field-effect transistors (FETs) manage the current between source and drain 
through an electric field created by the gate. The electric field alters the shape of 
the conducting channel. In the case of CNT-based field-effect transistors 
(CNTFETs), the semiconducting SWCNT becomes conducting as soon as a certain 
voltage, and hence an electric field, is applied. Due to the small size of SWCNTs, 
it is possible to reduce the size of FETs and to build up CNTFETs that can switch 
1000 times quicker than FET processors today [23]. 

The continuous miniaturization of silicon-based integrated circuits requires new 
vertical interconnects (VIAs) that can carry larger current densities. The 
mandatory vertical growth of CNTs can be realized by chemical vapor deposition. 
An additional application for CNTs is to act as horizontal interconnects. Also, the 
horizontal growth of CNTs can be realized by CVD [24]. A significant problem for 
applications in nanoelectronics is the band gap dependence on the CNT’s chirality. 
Today’s fabrication techniques generate a mixture of metallic and semiconducting 
CNTs. For industrialization, the growth of SWCNTs of a specific type, in a defined 
direction, and at a defined position, is required. 


7.2.5.4 AFM Cantilever Tips 

Atomic force microscopy has become an important technique in a wide range of 
research areas due to its ability to image and characterize nanoscale structures. The 
quality of information strongly depends on the size and shape of the AFM probes 
used. Commercially available AFM probes are usually made of micromachined 
silicon cantilevers with integrated pyramidal tips. Such tips have a typical radius of 
about 10 nm. Conventional AFM probes are, therefore, unable to penetrate high- 
aspect-ratio structures and to exactly profile surfaces with a complex topography. 
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Figure 7.11. Comparison of standard AFM tips and CNT-based supertips 


CNT-based supertips with superior characteristics (stability, resolution, 
lifetime, efc.) can overcome the limitations of micromachined silicon tips 
(Figure 7.11). Supertips, however, cannot only be used as the main sensing 
component ultimately responsible for the quality of AFM imaging. Their extremely 
small size and their high conductivity also make them suitable to function as ultra 
fine nanoelectrodes able to electrically contact nanoscale structures in 
nanoelectronics. 
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7.3 Characterization of CNTs 


In order to realize the above-mentioned applications and to make use of carbon 
nanotubes in nanotechnology products, a complete and correct charaterization of 
carbon nanotubes must be achieved. The production of carbon nanotubes in a 
controlled way, in large amounts, and with well-defined properties, encounters 
problems that remain to be solved. First of all, it is essential to identify all the 
physical properties of CNTs. A multitude of techniques are available to fully 
characterize carbon nanotubes. An overview of the existing characterization 
techniques will be given in the following section. 


7.3.1 Characterization Techniques and Tools 


Many properties of nanotubes are directly influenced by the way in which the 
graphene sheets are wrapped. The most important properties of carbon nanotubes 
are: 


length and diameter of the CNT, 

distinction between metallic and semiconducting SWCNTs, 
appointment of the chirality of SWCNTs (zigzag, armchair, or chiral), 
measurement of sample purity and alignment of CNTs, 

features of MWCNTs (outer and inner radius, intershell spacing). 


To determine all these properties, different characterization methods can be used 
[2]. As mentioned in Section 7.1, the most common tools and methods are: 


transmission electron microscopy, 

scanning tunneling microscopy, 

atomic force microscopy, 

scanning electron microscopy, 

spectroscopy (photoluminescence, X-ray, photoelectron, infrared, and 
Raman), 

e neutron and X-ray diffraction. 


7.3.1.1 Microscopic Characterization Methods 

TEM is useful for the characterization of the structure of MWCNTs. For example, 
the outer and inner radius, the number of layers, and the intershell distance of an 
MWCNT can be obtained [25]. High-resolution transmission electron microscopy 
(HRTEM) offers an even higher resolution than TEM, so that the atomic structure 
of CNTs becomes visible. For example, the nucleation point of an SWCNT or the 
atomic structure of both the catalyst particles and the SWCNTs can be imaged. By 
applying inverse fast Fourier transformation to HRTEM images, the chirality of 
SWCNTs can be determined [26]. 

STM is based on the nanoscopic phenomena of the quantum tunneling effect. 
STM images can resolve both the atomic structure and the 3D morphology of a 
CNT and mirror the convolution of the STM tip shape and the sample surface. 
Furthermore, the chiral angle of nanotubes can be determined by the analysis of 
STM images [27]. 
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As described in Chapter 1, AFM can be used both as an imaging and man- 
ipulation tool [28,29]. AFM-based measurement of the Young’s modulus of 
SWCNT bundles can be achieved. SWCNT bundles are deposited across the pores 
of a polished alumina membrane. Afterwards, an AFM probe is used to bend the 
bundles at their centers and to measure how the deflection, which is inversely 
proportional to the Young’s modulus, varies with the applied force [30]. 

As an advantage, images taken by an SEM can be recorded almost in real-time 
(Chapter 4). The geometric data of MWCNTs can be extracted from SEM images 
(Figure 7.12), and moreover, the relatively huge vacuum chamber allows the 
analysis of large samples and provides enough space for robotic nanomanipulation 
systems. 
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Figure 7.12. SEM micrograph of MWCNTs for measuring the geometric dimensions. 
MWCNT diameter is about 50 nm and length 1 um. 


7.3.1.2 Spectroscopic Characterization Methods 

For analyzing the different CNT properties, various spectroscopic characterization 
methods are available. One of these methods is the so-called photoluminescence 
spectrocopy. For this, individual SWCNTs are needed. The nanotubes are excited 
by laser light, and a close infrared E}; emission of the SWCNTs can be measured. 
Using this method, the conductive nature (semiconducting or metallic), the 
geometry, and the diameter are accessible. In the case of semiconducting 
SWCNTs, the gap energy is directly related to the chirality and approximately 
proportional to the tube diameter [2]. 

X-ray photoelectron spectroscopy (XPS) can be used to investigate nitrogen 
doping effects on the chemical structure of MWCNTs. By controlling the 
NH;/C,H) flow ratio during the CVD process, different nitrogen concentrations are 
obtained. According to XPS, the increasing nitrogen concentration leads to an 
increase of the N-sp? C bonds [31]. 
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Infrared spectroscopy can be applied to determine the symmetry of SWCNTs. 
There are 7-9 infrared active modes in SWCNTs, which depend on the symmetry: 
chiral, zigzag, and armchair. Also, impurities remaining from the synthesis of 
carbon nanotubes can be determined. For some applications of CNTs, molecules 
are attached to the surface of CNTs, in order to functionalize them. Infrared 
spectroscopy offers a way to characterize these functionalizing molecules [32]. 

Raman spectroscopy is one of the most powerful characterization techniques. It 
offers a fast and non-destructive analysis without sample preparation. An SWCNT 
sample is excited by laser light and a characteristic Raman spectrum is recorded. 
The so-called radial breathing mode (RBM) is directly dependent on the CNT’s 
diameter. The symmetry (zigzag, armchair, or chiral) and the conductive nature 
(semiconducting or metallic) are reflected in the position, width, and relative 
intensity of the characteristic G-line [2]. 


7.3.1.3 Diffractional Characterization Methods 

X-ray diffraction is not useful for the differentiation of microstructural details, but 
it can be used to determine the sample purity, the number of layers, the interlayer 
spacing, and the alignment of CNTs. The intensity and width of the characteristic 
peak in a typical X-ray diffraction spectrum is related to the number of layers and 
the alignment of the CNTs. The presence of catalyst particles (Co and Mo) 
remaining from the fabrication process can also be detected [2]. 

Neutron diffraction can be applied for the determination of structural features 
of CNTs, such as bond length and possible distortion of the hexagonal network. 
Even for the smallest diameters, a separation between armchair, zigzag, and chiral 
CNTs is possible. Moreover, the adsorption of He atoms on an SWCNT bundle can 
be analyzed [33]. 


7.3.2 Advantages of SEM-based Characterization of CNTs 


In Section 7.3.1, an overview of the existing characterization tools and methods 
was given. Having in mind the title of this book, we will now refocus on the 
microrobot-based handling and characterization of CNTs within the SEM. 

In order to observe nanoobjects and especially CNTs, the use of an SEM is 
necessary to overcome the limited angular resolution of optical microscopes. The 
developed SEM image processing (Chapter 4) allows for real-time SEM imaging 
and the recognition and tracking of nanoobjects using SEM images. Additionally, a 
3D imaging system is available to determine depth information and thus the full 
3D pose of nanoobjects (Chapter 5). 

Besides the possibility of imaging objects with sizes down to some nanometers, 
the SEM offers a relatively spacious vacuum chamber compared to other electron 
microscopes such as TEM or HRTEM. SEM therefore is the ideal type of micro- 
scope for developing and installing microrobot-based handling and characterization 
systems (Chapters 1 and 2). 
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7.4 Characterization and Handling of CNTs in an SEM 


Due to the remarkable physical properties of CNTs they are qualified for many 
applications, especially in nanoelectronics. However, the realization of these 
applictaions requires the handling and characterization of individual CNTs. SEM- 
based nanohandling robot stations can be used for the prototyping of 
nanoelectronic devices. The big advantage of visualizing the handling and 
characterization processes of CNTs has led to the development of various micro- 
and nanorobotic systems for the use within the SEM. 

For example, [34] developed a nanolaboratory, a prototype nanomanufac- 
turing system based on a 16-DoF nanorobotic manipulation system for the 
assembly of nanodevices with MWCNTs. This system can use up to four AFM 
probes as end-effectors. The position resolution is of subnanometer order, and 
strokes are centimeter scale. Jn situ property characterization, destructive 
fabrication, shape modifications, and CNT junctions have already been presented. 
However, picking up CNTs with an AFM probe and by the use of dielectrophoresis 
or EBiD is reported to be difficult for strongly rooted CNTs and for transferring 
CNTs to another structure. 

[35, 36] use a hybrid AFM/SEM system for the placement of an individual 
CNT onto a predetermined site on a microelectromechanical structure (MEMS). A 
single CNT is then retrieved with the AFM tip from a CNT cartridge using van der 
Waals forces between tip and CNT. Afterwards, the CNT is transferred to the 
desired site on the MEMS and then placed across a gap between a stationary 
structure (reticle) and a thermal actuator (pointer). The CNT is welded to the reticle 
and pointer by electron-beam-induced deposition of carbonaceous material. 
Considerations about electrical resistivity versus mechanical strain measurement 
on a CNT are then discussed. 

Another system for three-dimensional manipulation and mechanical as well as 
electrical characterization has been realized by [37]. Here, a custom piezoelectric 
vacuum manipulator achieves positional resolutions comparable to scanning probe 
microscopes (SPM), with the ability to manipulate objects in one rotational and 
three linear DoF. This system is used for the manipulation of CNTs in order to 
explore their electromechanical properties under real-time SEM inspection. CNTs 
are stressed while measuring their conductivity, and CNTs are attached to AFM 
tips such that forces applied to the CNTs can be determined by the cantilevers’ 
deflection. In addition, bending, kinking, breaking, and bundling of CNTs have 
been observed. 

The nanomanipulator-assisted fabrication and characterization of CNTs inside 
an SEM has been reported by [38]. Here, two nanomanipulators are installed which 
can travel about 20 mm with a minimum increment of 1 nm. Electrochemically 
etched polycrystalline tungsten wires (STM tips) were used as end-effectors. This 
setup was used to construct a CNT transistor, to attach an MWCNT onto an AFM 
tip, to observe the elongation of a CNT, to modulate the shape of a CNT for tool 
fabrication (CNT hook), and to show in situ characterization of the electrical 
breakdown of MWCNTs. The latter was mentioned to provide a new method for 
characterizing the nature of conductivity of MWCNTs. It is worth mentioning that 
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this method is a destructive one, so that the characterized CNTs cannot be used for 
further assembly. 

Progress towards the nanoengineering of CNTs for nanotools has been made 
by [39]. The authors use AC electrophoresis for aligning MWCNTs at a knife- 
edge, forming a CNT cartridge. These MWCNTS are then transferred from the 
CNT cartridge onto a substrate in an SEM, in order to assemble nanotube AFM 
tips and nanotube tweezers that can operate in SPM. The fabricated nanotube AFM 
tips have been tested in AFM observation of biological samples and traces with 
deep crevices. The resulting nanotube tweezers are installed in an SPM to 
manipulate SiO, particles and CNTs. Moreover, the electron ablation of an 
MWCNT has been realized to adjust its length and to sharpen the MWCNT, having 
its inner layer with or without an end cap at the tip. The measurement of the sliding 
force for the inner layer with diameter 5 nm showed that the force amounts to 
approximately 4 nN, independent of the overlapping length. 

The nanomanipulation of nanoobjects and CNTs in an SEM is also described in 
[40]. The authors present a sensor-based manipulation and processing system and 
demonstrate the gripping of micro-sized powder particles and the attachment of 
CNTs on tips for atomic force microscopes. 

Great progress in the field of nanoengineering has been achieved by the 
Department of Micro and Nanotechnology (MIC) of the Technical University of 
Denmark. Besides the electrical and mechanical characterization of CNTs [41, 42], 
the development of microfabricated grippers [43] has advanced the pick-and- 
place nanomanipulation of nanowires and nanotubes [44,45]. Moreover, the 
integration of CNTs into microsystems with electron beam lithography and in situ 
catalytically activated growth has been realized [46]. 

Several other methods for the characterization and handling of CNTs have been 
demonstrated. For example, the manipulation inside a TEM [47,48] is 
advantageous because of the availability of diverse characterization tools down to 
atomic resolution (Section 7.3.2.1). However, TEM offers a very small vacuum 
chamber that is not easily accessible for robotic systems. 

Another possibility is to use AFM or STM to manipulate CNTs [28, 49], 
nanoobjects [29, 50], or even single atoms [51, 52]. The big disadvantage of such 
SPM-based techniques is that the handling task can be carried out in only two DoF 
and must be performed “blindly”, since during the manipulation, the SPM tip is not 
available for imaging the scene, so that SPM images can only be acquired before 
and/or after the manipulation. 

Contact-free manipulation techniques have also been developed. Firstly, 
multidimensional manipulation of CNTs with optical tweezers has been reported 
[53], secondly, dielectrophoresis has been applied to the assembly of MWCNTs 
[54], and thirdly self-assembly has been used for the assembly of SWCNTs into 
hierarchical structures with controlled nanotube orientation [55]. However, for 
these contact-free methods, the CNTs must be dispersed in a liquid solution. 

Recapitulating, SEM-based nanohandling robot systems are the most promising 
method for a controlled and reliable three-dimensional manipulation and character- 
rization of CNTs and thus for assembling them into new prototypical nano- 
structures and -devices. 
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7.5 AMNS for CNT Handling 


Following the generic concept of the automated microrobot-based nanohandling 
station (AMNS) introduced in Chapter 1, a robot station was developed that can be 
integrated into an SEM and can use different end-effectors for the non-destructive 
characterization and reliable three-dimensional handling of CNTs. Either the tip of 
a piezoresistive AFM probe for the mechanical characterization of CNTs or an 
electrothermal nanogripper [43] for CNT handling is applied. 


7.5.1 Experimental Setup 


A mechanical drawing of the experimental setup of the nanohandling robot 
station is shown in Figure 7.13. The setup is mounted on a base plate and contains 
two different manipulators. There is a three-axes micromanipulator MM3A [56] 
that is equipped with an end-effector holder and is used for the coarse positioning 
between end-effector and CNT sample. 


AFM probe or nanogripper CNT sample 








3-axes micromanipulator nanopositioning stage 


Figure 7.13. Mechanical drawing of the nanohandling robot station 


The micromanipulator offers a theoretically possible resolution of 5 nm for the 
two rotational axes and 0.25 nm for the linear axis. A nanopositioning piezo stage 
with three degrees of freedom carries the CNT sample. The resolution of the 
nanopositioning stage [57] is limited by the 16-bit D/A converter of the control 
module and amounts to 1.55 nm in closed-loop mode. 

For nanohandling inside the SEM, the whole setup is installed onto the 
positioning stage of a LEO 1450 SEM [58] (Figure 7.14). 


Characterization and Handling of Carbon Nanotubes 219 





Figure 7.14. The nanohandling robot station within the vacuum chamber of the SEM 


Figure 7.15 shows the overall station’s control system that is currently used for 
the teleoperated nanomanipulation of CNTs in the SEM. By operating the SEM 
stage, the whole setup is adjusted within the vacuum chamber, getting it into the 
SEM focus, so that the manipulation scene can be observed by using the SEM 
imaging software. After adjusting the SEM image, the  three-axes 
micromanipulator is used for the coarse positioning of the end-effector close to the 
CNT. For the fine positioning and thus the real manipulation of CNTs, the 
nanopositioning stage is controlled in order to move the stage in 1.55 nm steps. 
The stage controller delivers an encoder signal, giving the exact value of the 
stage’s displacement, so that the stage can be operated in closed-loop mode. The 
nanopositioning stage is operated by control software written in LabVIEW and the 
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Figure 7.15. Control system architecture for telemanipulation 
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7.5.2 Gripping and Handling of CNTs 


The nanomanipulation, meaning the controlled gripping and handling of carbon 
nanotubes, is of great importance for nanotechnology. Therefore, new gripping 
tools have to be designed that are suitable for the handling of nanoobjects and 
CNTs [44, 59]. The electrothermal nanogrippers used for the CNT handling below 
have been provided by MIC, Denmark. The electrothermal gripper can be 
actuated in various modes by applying voltages to the different gripper beams [43]. 
The width of the gripper arms and the gripper opening, is 2 um. 

As mentioned above, the capability to observe the nanohandling process in 
real-time with high magnification is achieved by operating the nanohandling robot 
station within an SEM. A frequent nanomanipulation task is the gripping of a car- 
bon nanotube and placing it onto a TEM grid for further analysis and charac- 
terization in TEM, or placing it onto an electrode structure for electrical 
characterization. Currently, the gripping process itself is done by teleoperation. 
Automatic positioning of gripper and CNT sample will be realized using a special 
control system described in Section 7.6. Such automation of nanohandling 
processes will be the next step towards the integration of new, promising nano- 
materials into conventionally micromachined parts and thus the prototyping of 
nanoelectronic components. Figure 7.16 shows the process flow of gripping an 
MWCNT grown on silicon substrate by chemical vapor deposition. 





c HE 


Figure 7.16. Gripping sequence of an MWCNT: a. coarse and fine positioning between 
gripper and CNT has been realized; b. gripper actuation; c. removing the CNT from the 
substrate; d. reopening the gripper (in collaboration with MIC) 
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The arrays of MWCNTs have been provided by the Engineering Department, 
University of Cambridge, United Kingdom [20]. The diameter of the handled 
MWCNT is about 200 nm and the length is about 6 um. In Figure 7.16a, the coarse 
and fine positioning of the nanogripper close to the CNT has been realized in order 
to grip the CNT. By applying a voltage to the nanogripper, the gripper is actuated 
in order to grab the CNT (Figure 7.16b). By moving the CNT with the nano- 
positioning stage and keeping the position of the nanogripper fixed, the CNT is 
removed from the substrate and can be positioned in three dimensions 
(Figure 7.16c). After reopening the nanogripper, the CNT sticks to the right gripper 
jaw due to adhesion forces (Figure 7.16d). The controlled put-down of the CNT 
on a well-defined location, like a TEM grid or a electrode structure, will be the 
next step of exploration. Electron-beam-induced deposition (Chapter 10) will be 
used to fix the CNT on the target structure in order to overcome these adhesion 
forces. 


7.5.3 Mechanical Characterization of CNTs 


The nanohandling robot station can also be used to perform deflection 
measurements of MWCNTs, in order to calculate their Young’s modulus Æ. 
Figure 7.17 shows the principle of the bending experiments using a tipless piezo- 
resistive AFM probe. 





c d 


Figure 7.17. Sequence of piezoresistive AFM probe-based deflection of an MWCNT: a. 
coarse and fine positioning between AFM probe and CNT has been realized; b. beginning of 
CNT bending; c. maximum CNT bending; d. coming full circle 
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In Figure 7.17a, the coarse and fine positioning of an MWCNT and an AFM 
probe has been realized. The MWCNT used for the deflection experiments has a 
length of 8.8 um and a diameter of 370 nm at the bottom and 160 nm at the top. By 
moving the nanopositioning stage and keeping the AFM probe fixed, the MWCNT 
counteracts the AFM probe and bends (Figure 7.17b-c). The deflection of the CNT 
is measured with the help of the position sensor of the nanopositioning stage in 
closed-loop mode, and the bending force is measured by the piezoresistive readout 
of the AFM probe deflection and on the basis of the following AFM probe 
calibration. 

The first step of AFM probe calibration is to measure the geometric 
parameters of the AFM probe beam at different points using the SEM. The average 
of these values can be used to calculate the probe’s stiffness. The geometric 
parameters (length /, width w, and thickness f) of the piezoresistive AFM probe for 
these measurements are: 


l = 507.5um, w=157 um, t = 7 um. (7.8) 
The stiffness S of the AFM probe can be calculated by: 


Ew? 
S=——_, 7.9 
4P 1?) 


where E is the Young’s modulus of silicon and the parameters w, t, and / are the 
geometric values of the AFM probe. Thus, the calculated stiffness amounts to 
S= 17.08 N/m. 

The sensor signal of the piezoresistive Wheatstone bridge integrated into the 
AFM probe is a differential voltage, which has to be calibrated to the corres- 
ponding force. Calibration can be done easily by deflecting the AFM probe against 
a hard reference material such as a hard silicon specimen. By recording the output 
voltage and the deflection of the AFM probe, the calibration factor K that connects 
voltage and force can be calculated. Figure 7.18 shows the measured 
voltage-deflection curve during AFM probe calibration. A linear fit gives a 
voltage-deflection factor V of 3.76 mV/um. With this factor V and the stiffness S, 
the calibration factor K of the AFM probe can be calculated by: 


K= 


xja 


(7.10) 


The calculated stiffness of S= 17.08 N/m and the voltage-deflection factor of 
V =3.76 mV/um lead to a calibration factor of K = 4.537 uN/mV. By multiplying 
the (offset corrected) voltage signal obtained from the piezoresistive AFM probe 
by this factor K, the effective force F can be calculated. For the final CNT bending 
experiments, the deflection of the MWCNT is performed by moving the nano- 
positioning stage in an automatic ramp function. 
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Figure 7.18. Calibration curve for the piezoresistive AFM probe 


The associated force-deflection curves are therefore generated automatically. 
Figure 7.19 shows the measured force depending on the CNT deflection for the 
realized automatic CNT bending. The maximum CNT deflection is approximately 
1.5 um, with a resulting force of 2.5 uN. The correspondence between loading and 
unloading emphasizes the elastic behavior of MWCNTs. 
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Figure 7.19. Force-deflection curve for MWCNT bending 
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The slope of the linear region is evaluated as m=2.44 N/m. According to 
Equation 7.6 and Equation 7.7, the Young’s modulus of the MWCNT can be 
calculated by: 


64ml° 
E Saat’ (7.11) 
where / is the length of the MWCNT, given by the position of the nanotube having 
contact with the AFM probe, £E is the Young’s modulus, and d the diameter of the 
MWCNT. 

The multiwall carbon nanotube has different diameters at the bottom and at the 
top of d,=370 nm and d,= 160 nm. The Young’s modulus has been calculated 
for both values, and the mean value leads to E£ = 0.95 TPa, with /=4.17 um and 
m=244N/m. This presented result is in good conformity with theoretical 
calculations of the Young’s modulus of MWCNTs predicting values of about 
1.05 TPa [8, 9]. 


7.6 Towards Automated Nanohandling of CNTs 


The previous sections have introduced techniques and tools to handle and charac- 
terize CNTs. However, performing these experiments on a large number of CNTs 
is a very time consuming and tedious process. Even though manual handling might 
be sufficient to achieve good research results on physical material constants, it is 
not good enough for industrial purposes or material tests that require a large 
number of samples. For the industrial usage of CNTs, reliable and repeatable 
handling of nanostructures at high throughput is required, which is accomplished 
for macro objects by automated manufacturing and quality control. However, the 
single automation cells (i.e., setup and control) in nano- and microfactories need to 
be flexible enough to be reused for several different automation tasks. 


7.6.1 Levels of Automation 


Different degrees of automation can be considered. The lowest level is tele- 
operation, where all movements are performed by a human operator. It is the 
responsibility of the operator to avoid hitting objects, to perform the operation in 
the right order to accomplish the assembly goal and to literally move the actors 
from one to another position. 

The next level of automation is semi-automation. Low-level closed-loop 
control is available for some, or even all operations. A human operator can trigger 
a movement. For example an actor should move into a predefined parking position 
on a single operator command. The execution of the task is handled by the 
responsible closed-loop controller automatically. A human operator is responsible 
for selecting the correct input sensors and sensor models. In addition, the operator 
needs to check whether a closed-loop goal has been achieved or not and resolve 
possible errors. 
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The highest level of automation is full automation. At this level, the full 
assembly sequence can be run without human intervention, and all assembly steps 
can be separately performed as closed-loop operations. A full automatized 
assembly sequence requires high reliability of all single subtasks and knowledge of 
the assembly situation before and after the execution of every single task, in order 
to be able to handle errors. Collision avoidance is achieved by the abstract planning 
of end positions for every step. These end positions get split into intermediate 
steps, starting from the current position such that the robot cannot collide with 
other objects. The intermediate steps introduced by the high-level controller need 
to avoid collisions, and they are executed one by one by the corresponding low- 
level controller (LoLeC). 


7.6.2 Restrictions on Automated Handling Inside an SEM 


Automation in microrobotics encounters many of the problems that have been 
thoroughly studied for decades in the domains of industrial robotics and auto- 
nomous service robotics. Some of the problems are, e.g., collision avoidance in 
path planning, error handling due to uncertainty of operations [60], or timing 
constraints which need to be met for successful automation. 

However, there are some environmental challenges while operating in the 
vacuum chamber of an SEM. It takes several minutes to generate a high vacuum, 
which is a serious time constraint. Therefore, all automation cell parts, or even a 
complete manufacturing line, need to be inside the SEM before the operation starts, 
including tools and objects to be handled. The workpieces need some kind of a 
depot, and so do the end products. Tools, sensors, and objects have to be vacuum- 
compliant. Depending on the kind of manipulation, the materials need to be 
selected such that no contamination of the preprocessed workpieces (e.g., wafers) 
and the clean room can occur. 

Physics in microrobotics works the same way as in macrorobotics. There are, 
however, some differences. In contrast to large-scale objects, it is harder to release 
an object than to grip it. The reason for this “sticky finger” effect [61] is that 
adhesive forces are stronger at the scale of the gripper jaws and samples than 
gravity.' The released objects most often do not settle directly below the gripper. 
They jump around unpredictably, attracted by local surface forces. Therefore, pick- 
and-place operations need to be carefully planned. There is as yet no perfect 
solution for this problem. However, one possible solution is to reduce the contact 
area between gripper and object by reshaping the gripper or by stripping off the 
object at an edge. Another possibility that gives more control to the surface end 
position of the object is to use EBiD. One end of the gripped CNT is fixed to the 
specimen holder by the deposited material of the EBiD process, just before 
releasing it from the gripper (Chapter 10). 





! In fact, gravity is the least significant force below a scale of 20 um compared to van der 


Waals, electrostatic, and capillary forces. 
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The sizes of both the tools and objects also introduces some automation 
challenges to sensors. The AMNS setup presented in Section 2.6 introduced SEM 
images as the most important source of control information due to their high 
scanning rate and resolution. For automation, SEM vision fulfills several important 
tasks. One vision task is real-time position tracking of end-effectors using the 
SEM. In closed-loop control, the image acquisition and processing time is one of 
the major constraints. If the images are generated too slowly, the robot may 
overshoot its position or, even worse, never reach it. High scanning rates can only 
be achieved with the tradeoff of noisy image data in an SEM (Chapter 4). 

Another important vision task is to obtain three-dimensional information on 
the automation environment. Without the additional depth information, one can 
only grip an object using a trial-and-error approach: “Move the gripper and close 
it; test if a CNT has been gripped, if not reopen gripper and retry”. Collision 
avoidance also needs reliable depth information. How to add the third dimension to 
SEM images is described in more detail in Chapter 5. As a last task, quality control 
through SEM vision is important. Many actors lack sufficient feedback. SEM 
images are one way of gaining knowledge about the outcome of an operation. 
Here, vision acts as a classifier for questions like “Was the last placement ope- 
ration successful?” and “Is the object at the right position? ”. 


7.6.3 Control System Architecture 


The AMNS setup for handling TEM lamellae (Section 2.6) is the first imple- 
mentation of AMNS control architecture for automated nanohandling. However, it 
has some major limitations that make modifications necessary for the presented 
CNT characterization tasks. The old architecture cannot handle linear actors and 
consists of one low-level controller only, which does not scale up to setups with a 
larger number of robots working in parallel. In addition, integrated robot sensors 
cannot be used, since the low-level controller is not designed as a sensor program. 
Therefore, a new system architecture has been developed for the CNT handling. 

The new system consists of high-level control, sensor, vision, and several low- 
level control servers (Figure 7.20) following a “black box design” [62]. Every 
server offers an individual service, which is defined by a public interface. There- 
fore, every component in this modular control system can be easily replaced or 
updated independently of the other components. 

Visual feedback is provided by the vision server. It is responsible for collecting 
images from different sources and extracts position and orientation data (poses) of 
tracked objects in real-time. These poses are transmitted to the sensor server. All 
sensor data (e.g., pose, touchdown, force, or temperature) are collected by the sen- 
sor server, which is supplied by different sensor service programs (e.g., vision 
server). These data are provided to low- or high-level control. It therefore repre- 
sents an abstraction layer for sensors and builds a common interface for further 
processing. The different sensors may acquire their data at different update rates. 

In this architecture, there is one low-level control server for every single actor. 
Every individual low-level control server retrieves the data needed for closed-loop 
control from the sensor server. For this purpose, the sensor server provides two 
data retrieval models, a data push and a data pull model. The data push model 
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sends new data to the requesting low-level controller if it is available, limited by a 
maximum rate threshold. Low-level controllers using the pull model periodically 
ask the sensor server for new data. There is a uniform interface for commands from 
high-level control. The advantage of this lean low-level control server approach is 
that single servers can be easily distributed among several PCs. Their good main- 
tainability is another important feature. Furthermore, it is easy to include different 
robot platforms (e.g., stationary and mobile), because only the internal structure 
of the low-level controller has to be changed. On the high-level control level, still 
only poses have to be submitted through the command interface. Actors with 
internal sensors are included in this architecture in a way that the low-level control 
server provides the internal sensor readings to the sensor server and receives high- 
level commands. 
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Figure 7.20. Software architecture connection chart. Rectangles are servers, and ellipses are 
hardware components. The automation module is part of the high-level control server. 


The high-level control server (Figure 7.21) processes an automation sequence 
or receives teleoperation commands from a user via graphical user interface (GUI). 
Input information is translated into low-level command tokens called tasks and 
steering signals for the selection of certain sensors. High-level control is respon- 
sible for automation, path planning, error handling, and the parallel execution 
of tasks. 

In order not to adapt the high-level control server to every change in an 
automation sequence, a script language has been developed, which is described in 
more detail in Section 7.6.5. These scripts are interpreted by the high-level control 
server and then mapped to low-level control tasks and steering signals. Low-level 
control tasks can either be closed-loop for positioning or open-loop (e.g., gripping 
objects) tasks. The latter have to be monitored to check the operation results. 

Every task has a defined set of pre- and postconditions. Preconditions need to 
be met before a task can be executed. The postconditions do hold after successful 
task completion. High-level control decides, based on required resources (e.g., 
sensors and actors) and pre- and postconditions, whether two consecutive tasks can 
run in parallel. These postconditions should not be contradictory. If resource 
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conflicts arise, a barrier approach is taken, so that all parallel tasks are terminated 
before automation of the resource-critical task starts (Figure 7.22). For SEM auto- 
mation, these resource-critical tasks are often tasks where the SEM is used as a 
tracking sensor for two objects at different heights, or where the SEM acts as 
sensor and actor at the same time (e.g., EBiD and a concurrent positioning task). 
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Figure 7.21. High-level controller consisting of automation, path planning, execution, and 
teleoperation unit 
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Figure 7.22. Schematic view on an arbitrary automation sequence. Parallel tasks are 
terminated before resource critical tasks get started. 


Path planning can be done either during the automation sequence design or 
online. Online path planning is suitable for simple automation sequences, but it 
requires more knowledge about the work scene for collision avoidance. For online 
path planning, only goal positions would be included in the automation sequence, 
and the high-level controller has to compute intermediate positions and low-level 
control tasks. At the current development stage, only offline path planning is 
supported, which is sufficient in most cases, but delegates more responsibility to 
the designer of the automation sequence. 

One of the major problems in micro- and nanorobotics is the high error rate of 
single automation tasks. Due to environmental and scale effects, operations that 
are simple in macro automation (e.g., positioning) have a high error rate. There 
might be endless positioning attempts because the low-level control error threshold 
is too low, due to changes in humidity or temperature. 
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As in macro automation, the error of a sequence £, is the multiplied error 
rate of every single automation task £, : 


Ex, =1- || (0-8). (7.12) 


tetasks 


For example for a sequence consisting of seven tasks, each having an error rate of 
10%, the error rate of the automation sequence is higher than 50%. 

The first step in dealing with these high error rates is to achieve reliable error 
detection, which is the task of the feedback unit in the high-level controller. This 
unit receives all error conditions raised by components of the system (e.g., a 
LoLeC) and presents them to the right abstraction level. One possibility for dealing 
with such error situations would be to jump back to an earlier task of the auto- 
mation sequence. The preconditions of the task need to meet the current system 
state. 

The proposed system architecture uses the Common Object Request Broker 
Architecture (CORBA) as the communication framework, which is object-orient- 
ed-middleware defined by the Object Management Group (OMG) and which is 
platform- and language-independent. There are several implementations of the 
CORBA standard, including some for real-time applications. CORBA’s Interface 
Definition Language (IDL) is used as the common language for all communication 
interfaces between different network components. Therefore, the servers and 
clients in the system architecture merely need to implement the required IDL 
interfaces to be able to communicate with each other via simple method calls. A 
major advantage of IDLs is that they can be translated into different programming 
languages, which enables heterogeneous software design. The communication 
overhead in a closed-loop cycle has been evaluated as low enough (< 5 us) in a 
local fully switched Ethernet network. The limiting time factor remaining is, 
therefore, the image acquisition time. 

The interfaces of the client-server architecture are designed to provide asyn- 
chronous communication. All control commands return unique process IDs, so that 
delayed control feedback can be matched with the corresponding command. This 
part of the control feedback is crucial for successful, reliable automation. The 
system also has a common time base enabling low-level control servers to decide 
whether sensor data is outdated or not. The synchronization is periodically trig- 
gered by a master clock. 

Every network component is designed in such a way that it can be run on 
different PCs, which makes the system fairly flexible. The distribution of low-level 
control servers is especially useful for control cycles that run in parallel, as the 
distributed controllers do not compete for the same PC hardware resources. The 
control architecture may contain several sensor servers for different data to 
overcome possible data acquisition bottlenecks. It is possible to organize the sensor 
data traffic, shaping a minimum update interval, so that the inbound traffic of any 
single sensor server can be controlled. 

The problem of keeping several low-level control programs maintained is 
tackled by common low-level server templates, which are available for different 
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actor types. This also enables rapid design and the integration of new actors. 
Similar templates will also be provided to the sensor data acquisition server. 


7.6.4 First Implementation Steps 


The general system architecture presented in the last subsection can easily be 
tailored to meet the requirements of the nanohandling robot station presented 
earlier in this chapter (Section 7.5.1). The single system components are illustrated 
in Figure 7.23. For this setup, three low-level control programs need to run in 
parallel, one for each actor group. Our low-level controller for the nanopositioning 
actor [57] implements the high-level control interface as well as the provider 
interface of the sensor server. This is due to the fact that the actor has an integrated 
sensor. This sensor data is made accessible to high-level control through the sensor 
server. The closed-loop cycle, of course, is performed directly inside the LoLeC. 

After defining the system components, the modeling of a suitable automation 
sequence for a given handling task can be performed in three steps. Firstly, the 
hardware setup has to be defined according to the requirement analysis. Then, 
robot-based process primitives and their pre- and postconditions have to be defined 
(e.g., move robot to target position if it is in range of the vision sensor). Finally, an 
automation sequence has to be found, which meets all pre- and post-conditions of 
the process primitives, avoids collisions, and eventually accomplishes the auto- 
mation task. Additional constraints, as, e.g., executing time, can be taken into 
account as well. 
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Figure 7.23. Tailored control architecture for the experimental setup in Figure 7.13. LoLeC 
PI acts as low-level controller for high-level control and provides actor-integrated sensor 
information. 


A flexible script language has been developed, following the script-based 
approach of Thompson and Fearing [63]. The different commands of the language 
are the process primitives themselves. They are implemented as subclasses of a 
common task base class. This avoids reimplementation of common concepts such 
as error handling or message protocols. Composition and iteration are provided as 
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general language constructs. Based on this script language, arbitrary automation 
sequences on the predefined operators can be defined (Figure 7.24). 


lift (S_TOUCHDOWN_ DIST); 
move (EC, E WORKING POS); 
grip (TRUE); 


lift (S_ SECURITY DIST); 





move (S_DROP_ POS); 


Figure 7.24. Automation sequence that lifts the specimen holder, moves the robot into the 
focus of the electron beam, and grips an object 


A sequence file gets interpreted and executed by the high-level controller. In 
this way, the automation sequences can take advantage of the underlying closed- 
loop control. The concept enables rapid development of different automation 
sequences for the same set of process primitives, while the high-level control 
program remains unchanged. 

The language design has been chosen with regard to future application of 
planning algorithms such as Metric-FF [64] to find the optimal automation 
sequence for a given automation task. Error handling can be performed online as a 
local refinement of the offline plan. For error handling, reliable error detection is 
crucial. The outcome of single process primitives has, therefore, to be monitored to 
provide feedback to the high-level controller, reflecting the current system state. 


7.7 Conclusions 


Since the discovery of CNTs about 15 years ago, intensive research has been suc- 
cessfully carried out on their basics (Section 7.2) and characterization techniques 
(Section 7.3). Nevertheless, another couple of years will be required to make use of 
CNTs in nanotechnology products. The integration of individual CNTs into 
micro- and nanomechanical systems will be one of the key issues. To achieve this 
objective, SEM-based microrobotic handling systems have proven to be a 
promising technique for the manipulation and characterization of individual CNTs. 

An overview of the existing systems and the state-of-the-art has been given in 
Section 7.4. Most of the existing systems merely allow for destructive handling and 
characterization of CNTs. Thus, the characterized CNTs cannot be used for further 
assembly of prototypical CNT-based devices. 

The nanohandling robot station presented in Section 7.5 permits three-dimen- 
sional handling and non-destructive characterization of CNTs. As for pick-and- 
place manipulation of CNTs, the main problem of parasitic adhesion forces acting 
between CNT and gripper jaw has to be solved. Therefore, special handling 
strategies will be developed, and/or EBiD will be used to overcome the adhesion 
forces and to allow a defined put-down. Non-destructive mechanical characte- 
rization was presented in Section 7.5.3. Future research activities will also focus on 
the realization of a non-destructive electrical characterization of CNTs. This would 


232 Volkmar Eichhorn and Christian Stolle 


lead to a non-destructive mechanical and electrical in situ characterization of 
CNTs, so that the characterized CNTs, with their well-known physical properties, 
can be used for further assembly of prototypical CNT-based devices. 

The progress towards automated nanohandling was presented in Section 7.6. A 
control system architecture was introduced, and first implementation steps were 
shown, including a high-level automation language. The system is capable of being 
adapted to the arbitrary hardware setups of AMNS. A major future task will be to 
identify and build a set of automated handling tasks that can be easily combined for 
arbitrary research and manufacturing tasks. 

From the authors’ point of view, the concept of automated microrobot-based 
nanohandling stations is the most promising concept to overcome the limitations 
suggested by fundamental research on CNTs and to allow for the industrial utili- 
zation of CNTs. Thus, the above mentioned in situ characterization, the three- 
dimensional handling, and the automation of particular manipulation and charac- 
terization processes have to be realized. Of course, bulk production cannot be 
reached by automated microrobot-based nanohandling stations, but this concept 
will enable the systematical prototyping of CNT-based nano-electro-mechanical 
systems. 
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8.1 Introduction 


This chapter will exclusively focus on the manipulation and characterization of 
biological cells by an atomic force microscope (AFM). Although other methods 
such as optical tweezers [1], dielectrophoresis [2], efc. exist, not only would 
detailed commenting on every branch of biohandling go beyond the scope of this 
chapter, but also most of these techniques are well established, while AFM—based 
characterization and manipulation is a strongly developing area. A brief 
comparison of AFM, dielectrophoresis and optical tweezer as manipulation and 
characterization methods for biological objects is given in Table 8.1. 

Since its development in the mid—eighties by Binnig, Quate, and Gerber [3], the 
potential of AFM for characterizing biological objects has quickly been realized, 
and it has become a valuable tool in biological studies ever since. An AFM 
combines several abilities that make it a very attractive instrument for high- (sub— 
nanometer) resolution applicatons, comparable with a scanning electron micro- 
scope (SEM). The ambient conditions, under which the samples can be investi- 
gated and which allow for appropriate physiological conditions, are similar to 
working beneath an optical microscope. There is also the option to measure 
different properties, such as elasticity, conductivity, or friction, beyond the topo- 
logical information. Because of the nature of the AFM, not only visualization, but 
also manipulation by its tip is an option. However, the ability to manipulate has 
limitations. Special care has to be taken that the scan—mode—dependent effect does 
not become a significant factor. 

Because of these abilities, the AFM makes the observation or manipulation of 
processes on a molecular to tissue level of living specimens in a physiological 
environment possible. The first measurements included the mapping of the 
elasticity of bone [4] and the imaging of the activation process of human platelets 
[5]. Along with the development of different measurement modes, for instance the 
dynamic mode, the possibilities of biological characterization also evolved. In this 
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way, the gating processes for intercellular communication could be imaged, and 
the electrical reaction of stimulated single hair cells in the inner ear could be 
recorded. The binding forces of receptor ligand pairs or the adhesion of cells, as 
required when leucocytes attach themselves to infected cells, can be measured. 
Most of these experiments would not be possible without the aid of an AFM and 
help to push the frontiers of research, e.g., exploring cytological functions or laying 
a foundation for pharmaceutical accomplishments. 

The following section will give an overview of AFM basics, with special 
emphasis on the relevant parameters for the characterization of biological material. 
A brief introduction to several biological matters of interest for AFM —based 
research is given as well, to provide a background for the technical terms used in 
the last section. Several applications of AFM-—based handling and characterization 
of biological material will be described, with an emphasis on current developments 
that trace the frontiers of current research. 


Table 8.1. Comparison of AFM, dielectrophoresis and optical tweezers as manipulation and 
characterization methods for biological objects 














AFM Dielectrophoresis Optical tweezers 

Measurable and pN -uN pN pN 

exerted forces 

Requirements for Immobilized Dielectric (must Dielectric, for 

the object have movable objects large in 

charges) comparision to the 

wavelength of the 
light transparency 
(act as lenses) 

Environment Vacuum, liquid, air Vacuum, liquid, air Vacuum, liquid, air, 


but surrounding 
medium must have 
smaller index of 
refraction than the 





object 
Interaction between Force between Electrical field, non— Electromagnetic 
method and object object and tip, contact waves, non— 


contact or non— 
contact, surface 
can be damaged 


contact, damaging 
due to heating 
possible 











Possibility to Movement, Movement, rotation, Movement, sorting, 

manipulate cutting, pushing sorting stretching, heating 

Measurable Elasticity, friction,  Polarizability Viscoelastic or 

characteristics conductivity, force elastic properties, 
measurements, force measurements 
topology, etc. 

Imaging ability yes no no 
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8.2 AFM Basics 


As several well-established books about AFM exist [6, 7], this section does not 
intend to explain an AFM in full detail. Nevertheless, an overview of its func- 
tionality, with special emphasis on handling of objects in a liquid environment, is 
given. 

The functioning principle itself is quite simple: the heart of an AFM is a 
cantilever with a very sharp tip (usual tip radius of around 10 nm), which scans 
the area of interest, reminiscent of a record player as concerning its functionality. 
Due to interacting forces between tip and sample, the cantilever will eventually 
bend, which is detected. From the forces working on the cantilever, a 
topographical image of the sample can be generated. For scanning the sample, 
either the cantilever or the sample has to be moved with nanometer accuracy, 
which is usually carried out by piezostacks. The area of interest is limited. 
Normally, custom scanners can cover an area from 90 x 90 um to 100 x 100 um 
[8-10]. The scanning of an image usually takes some minutes, depending on scan 
rate, scan size, and mode. Similar to the SEM, scan speed and image quality are 
inversely proportional. Therefore, other methods to speed up the imaging process 
have been the subject of research, including AFM cantilevers with several tips at a 
fixed distance, as well as current developments on "video rate" AFMs, which will 
be covered in Section 8.2.6. 

As already mentioned, a main characteristic of the AFM is that it can work 
under very diverse ambient conditions, from different liquids to vacuum. It is 
worth noting that each of these conditions requires different parameters of the 
measurement modes. For the handling of biological objects, the medium of choice 
will mostly be liquid, so that the cells can proliferate in a physiological environ- 
ment. Another characteristic of the AFM concerns sample preparation. Since the 
tip "feels" the surface, no complex preparation is needed, for example, the need for 
drying and coating with a conductive layer when using an SEM. However, the 
samples have to be immobilized in respect to the tip. 


8.2.1 Cantilever Position Measurement 


For the detection of the cantilever bending, different methods can be deployed. In 
general, stationary systems — as are commercial AFMs — usually use optical detec- 
tion methods, while mobile systems (such as autonomous robots) use detection 
methods based on self—sensing options. Only the most common method, that of 
laser beam deflection, and the robot-relevant method of piezoelectric or 
piezoresistive cantilevers, will be presented here. Other methods, such as 
interferometry, electron tunneling, or capacitance—based options (although the 
latter is widely used in purely indentation—oriented measurement setups or as 
displacement sensors for piezoelectric scanners) are omitted, as they are laborious, 
suffer from noise problems, or are unstable and thus will not be found in modern 
systems. 
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8.2.1.1 Optical: Laser Beam Deflection 

The most common position detection method, used for almost every commercial 
system, is based on optical means. A laser beam is reflected by the cantilever. 
Every bending of the cantilever changes the position of the laser beam reflection, 
which is detected by a two- or four—-segment photodiode. It is difficult to include 
this method in small and non-static systems, such as a mobile platform carrying a 
cantilever. Another drawback is the two-dimensional representation of a three— 
dimensional sample, which is not crucial when measuring forces in no more than 
two dimensions, for instance force—distance curves or mere imaging. It has its 
shortcomings, however, when trying to measure three—dimensional forces. Lateral 
forces can be detected, but whether a cantilever is bent because of 
forward/backward or down/uplifting forces cannot be specified by this detection 
method. 


8.2.1.2 Self-sensing: Piezoelectric and Piezoresistive 

The deflection of a cantilever can also be measured piezoelectrically/resistively by 
incorporating a piezoelectric/resistive material on the cantilever, which reacts 
with a specific potential difference or different resistance to a specific bending of 
the cantilever. A clever design of the resistors allows a cantilever to register the 
forces working on it in different directions, enabling even three—dimensional force 
measurements. Electrical noise can make this method more complicated to use, but 
a resourceful setup can make it a good alternative, especially for mobile platforms. 
Another problem can be the sensitivity, as these cantilevers have to be very thin or 
relatively very long, in order to be soft enough for contacting biological samples. 
While very thin cantilevers are difficult to fabricate, long cantilevers are noise— 
sensitive. 


8.2.2 AFM Modes 


Different modes can be used for image acquisition or the manipulation of different 
samples, all having their advantages and shortcomings in imaging or manipulating 
samples. To make use of the unique ability of the AFM to be able to work with 
living specimens at nanoscale resolution, the specimens are usually kept in a liquid 
suspension to generate a native environment. This atmospheric condition influ- 
ences the acting forces between specimen, substrate, and tip, making the 
measuring in liquid different from measurements taken in ambient air or even in a 
vacuum. 

In liquid, the acting force regime is characterized by small capillary forces and 
weak van der Waals forces. Additionally, the Q factor is lower in an aqueous 
medium, due to its higher density, than in air, thus dampening oscillations more. In 
the following sections, three common methods of measurement are described, with 
special emphasis on their use in an aqueous regime with soft specimens. 


8.2.2.1 Contact Mode 

During AFM development, the first imaging mode that was used was the contact 
mode. As the name implies, the cantilever tip is "in contact" with the surface of the 
sample, recording topography by the cantilever bending due to repulsive forces. 
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When scanning, the tip is dragged along the surface, which leads to obvious 
problems with softer materials, because of the lateral force applied. This mode can 
also be used for manipulation, e.g., cutting chromosomes. Two different methods 
can be used for measuring in contact mode. 

In the constant height method, the height of the cantilever in respect of the 
sample is kept constant. The bending of the cantilever due to variations in topology 
of the material is used for measuring the height. This method is obviously not very 
sensitive to the sample and is not suitable for very uneven samples, as the tip might 
lose contact, thus stopping the imaging or, conversely, crashing into the sample, 
which would lead to the destruction of either sample or cantilever. 

Alternatively, the constant force method can be used, where the force acting 
on the cantilever (and thus the bending) is kept constant. To ensure this, the height 
of the cantilever is modified while moving over the specimen. The variation of 
height, which is detectable by the voltage applied to actuate the piezo—scanner, 
which either moves the probe or the scanner, therefore gives the necessary 
topological information. This method is more sensitive to the sample 
and, furthermore, is much more capable of adapting to uneven surfaces. 

The different ambient conditions do not influence this measurement method at 
large scale, but inherent lateral forces make it problematic to use with soft 
materials, as the specimen can easily be moved or destroyed. Another problem is 
caused by the strong capillary forces in ambient air, which drag on the sample 
additionally to the lateral forces, when the tip moves in respect to it. When 
measuring in liquid, the capillary forces, at least, degrade substantially. Measuring 
soft samples is still possible in constant force mode. Cantilever force constants, 
however, should lie around 0.1 N/m, and preferably be even smaller. These very 
soft cantilevers exhibit another problem for as they are very sensitive to noise, the 
measurement setup has to be very well isolated from vibrations. 


8.2.2.2 Dynamic Mode 

In dynamic mode, the probe is oscillated near its resonance frequency above the 
sample. The relevant interacting forces are long—distance van der Waals forces, 
which influence and change the frequency of the cantilever. This mode can also be 
divided into two sub—modes, non-contact and intermittent. 

For non-contact measurements, the cantilever oscillates above the sample, with 
a driving frequency lying slightly above the resonance frequency of the cantilever. 
This is chosen because the frequency of the cantilever decreases when brought into 
the force regime of the sample. This also leads to a decreasing amplitude, as in 
non-contact mode any contact of tip and sample is undesirable. Alternatively, the 
oscillation can be driven by a magnetic field. For this purpose, the cantilever is 
covered with a magnetic material. This approach is called MAC mode. The 
cantilevers for this mode can have a smaller spring constant as the amplitude 
vibrations in MAC mode are smaller. 

The intermittent mode can be seen as a gentle combination of the two pre- 
viously described methods of contact and non-contact mode, where the cantilever 
tip periodically comes in contact with the probe. Here, the cantilever is driven with 
a frequency slightly below its resonance frequency, thus leading to an increasing 
amplitude when approaching the sample. In this way, the tip periodically comes 
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into contact with the sample. This method is often used for biohandling as is treats 
the sample much more sensitively, almost entirely avoiding disturbing lateral 
forces. As with the true non-contact method, the capillary forces, which can 
disturb the measurements, are not present when the sample is immersed in a liquid. 
When measuring in liquid, the otherwise disturbing capillary forces cease to exist, 
but additionally, the van der Waals forces this measurement method is based on are 
lowered (approx. 10 times lower than in air), along with the Q factor, resulting in a 
less sensitive system. The resonance frequency drops as well, to about a third of its 
value in ambient air. As a further drawback, the applied forces are not directly 
known, because the feedback is gained from the amplitude, not from the normal 
force. Both dynamic methods have the advantage of small (intermittent) or no 
(non-contact) lateral forces, thus being very gentle on the specimen. 


8.2.2.3 Lateral Force Mode 

This mode is a variation of the contact mode, where the cantilever is moved 
perpendicularly to its aligment for a line scan. Lateral torsion of the cantilever is 
detected. The cantilever can thus measure the friction of the sample surface, as 
well as its topology. Because of the obviously strong lateral forces, this method can 
only be recommended for relatively hard samples that are very well fixed to the 
substrate. 


8.2.2.4 Jumping Mode / Force Volume Mode and Force—Distance Curves 

The jumping, or force volume, mode is a series of force—distance curves evenly 
distributed over the desired specimen, coupled with feedback for approaching the 
sample in between each curve taken. The cantilever is brought into contact with the 
surface, with active feedback keeping it at the set point value. Upon reaching the 
set point, the cantilever is retracted with turned—off feedback, thus measuring the 
force—distance curve. At the maximum tip—sample distance, the tip is moved to the 
next measurement point, avoiding lateral forces on the sample. The force acting on 
the specimen is controlled by the set point. The characteristic shape of force- 
distance curves in distinct atmospheric conditions differs owing to the specific 
acting force regimes. In ambient air, the capillary forces are rather strong (Figure 
8.1a). Having approached the surface, the cantilever jumps into contact, because of 
the attractive forces acting between tip and sample. Upon retracting, the cantilever 
"clings" to the sample surface until the retractional forces overcome the adhesive 
forces. This point is called the snap—back point. These forces are strong enough to 
disturb or even to destroy sensitive specimens. For specimens covered with one or 
more additional layers (e.g., liquid layers are common), the part of the retraction 
curve where the cantilever still clings to the surface can be in the style of a step 
function. This is owing to the different snap—back points at which the cantilever 
breaks free from the current layer or specimen. While the capillary forces tend to 
be a problem in ambient air, they cease to exist in an aqueous solution, making the 
jumping mode a very sensitive method there. The curve in Figure 8.1b shows a 
typical force distance—curve in liquid, where the capillary forces on the cantilever 
are nearly non-existent. Thus, jumping mode allows the exertion of a controlled 
force on the specimen all the time, which is an improvement on the unknown 
forces applied in intermittent mode. 
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Figure 8.1. Force vs. distance, plotted for a: ambient air and b: liquid 


Because of the additional effort of the force—distance curves, this method is 
rather slow in comparison to both the contact and dynamic mode. However, it is 
easier to implement than the dynamic mode, since no dynamic options have to be 
integrated in hard— or software. As a bonus, adhesion and elasticity information 
can be gained at the same time as topological information. The cantilever 
specifications are similar to contact mode, needing soft cantilevers to avoid 
damage to the specimen. 


8.2.3 Measurements of Different Characteristics 


Depending on cantilever, tip, and mode, different characteristics of a sample can be 
measured. The capability of the AFM not only to conduct topographical mea- 
surement, but also to measure different characteristics, ranging from elasticity to 
conductivity to chemical composition, is established this way. As topological infor- 
mation, including height profiles and general spatial measurements, is rather self- 
explanatory, it will not be dealt with explicitly. 


8.2.3.1 Mechanical Characterization 

By means of force—distance curves, several, mostly mechanical, characteristics of 
a sample can be measured. This includes, for example, the Young's modulus, 
stiffness, and adhesion. 

When measuring the Young's modulus of a cell membrane or a biofilm, it has 
to be taken into account that the material itself will give way to the acting force. 
The force detected on the cantilever is, therefore, combined with the indentation 
depth: 


F =k-d(z)=k-(z-6) (8.1) 
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where F is loading force, k cantilever spring constant, d(z) cantilever deflection, z 
piezo position, 6 indentation depth. 

The curves are generally not as steep as indentation curves on a hard substrate 
such as silicon (Figure 8.2). For calculating the Young's modulus of the substrate, a 
model developed by Hertz and enhanced by Sneddon [11] is used: 


wee tan(a) 6”, (8.2) 


Pone z (1-v°) 


where F is loading force, E Young's modulus, v Poisson ratio, a half opening angle 
of conical tip, ô indentation depth: 





z-k-d(z):(1-v°) 
E= = (8.3) 
2-tan(a@)-(z-d(z)) 





With a measured force—distance curve, this equation leaves only the Young's mo- 
dulus Æ as a variable. Thus, Equation 8.3 has to be fitted to the linear part of the 
measured curve by the variation of E. 
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Figure 8.2. Cantilever deflection vs. piezo displacement, plotted for hard and soft surfaces 
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For the force—distance curves, sometimes the use of spherical tips is preferable 
to conical ones, due to a better defined contact area, more sensitive handling 
(damage by poking), and a smaller indentation depth. For spherical tips, the 
loading force is described by: 


4 E 1/2 63/2 
L R8, (8.4) 


paraboloid z 3 1 _y 
where F is loading force, E Young's modulus, v Poisson ratio, R sphere radius, 6 
indentation depth. 

The thickness of the material is another important parameter, as with very thin 
materials, the substrate below can influence the measurements, due to the inden- 
tation depth [12]. 


8.2.3.2 Magnetic Force Measurements 

Measurements of magnetic forces are conducted in non-contact mode by a tip 
coated with a ferromagnetic material. As the magnetic forces have a larger range 
than the forces exerted by the surface, the two forces acting on the cantilever can 
be distinguished. The measurement is conducted in two steps. First, a topological 
image is taken. In a follow-up scan, the cantilever is raised to a certain height 
(below 100 nm) where only the longer-ranging magnetic forces influence the 
frequency shift of the cantilever. 


8.2.3.3 Conductivity Measurements 

In this mode, topological and conductive information can simultaneously be 
gained. Conductivity measurements can be conducted in contact or non—contact 
mode. In contact mode, an AFM tip with a conductive coating is used. A voltage 
bias is applied between cantilever and substrate and from the resulting current 
flow, a spatially resolved conductivity image can be measured. This is of interest, 
for instance, in visualizing ionic processes across a cell membrane. 

For measuring the electrostatic characteristics of a sample, a voltage is applied 
between tip and sample, but the cantilever is not brought into contact while 
scanning the sample. The electrostatic forces acting between tip and sample 
influence its deflection, and thus charged domains can be detected. Another option 
for non-contact measurements is Kelvin probe force microscopy, in which a 
charged cantilever is scanned across a surface with applied potential. The potential 
is composed of an AC and a DC component. Potential differences of the DC part 
between cantilever and sample surface make the cantilever oscillate. By means of a 
feedback mechanism, the potential of the cantilever is then fitted to minimize the 
difference and thus the oscillation of the cantilever. As a result, the work function! 
of the sample surface can be calculated from the measurements. 


' The work function describes the minimum energy that has to be applied to remove an 
electron from inside of a solid to a point directly outside of the solid's surface. 
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8.2.3.4 Molecular Recognition Force Measurements 

By means of functionalized cantilevers, the chemical structure or intermolecular 
binding forces can be analyzed. The tip of the cantilever is coated with ligands or 
antibodies (Figure 8.3), which have a specific corresponding partner molecule in 
the sample. Usually, the sample is brought in contact with the tip (or vice versa, 
depending on the AFM setup) and, after reaching contact, is slowly retracted. The 
breaking of the intermolecular bonds can easily be measured by the snap—back 
point of the resulting force—distance curve. 
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Figure 8.3. Measuring ligand-receptor binding forces with a functionalized cantilever 


An improvement on this method has been developed [13], where an antibody 
was bound to the tip by a flexible part, allowing much better single binding of the 
respective molecules (Figure 8.4). The cantilever can be driven in dynamic mode, 
so that in addition to the binding force experiments and the recognition measure- 
ments, where the distribution of certain molecules in the sample can be visualized, 
topological measurements can be done at the same time. 
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Figure 8.4. Measuring ligand-receptor binding forces with a functionalized cantilever 
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8.2.4 Sample Preparation 


Although sample preparation is not as complex as for the SEM, which requires 
its samples to be dried and conductively coated, the sample has to be brought onto 
a substrate or otherwise immobilized. Different immobilization methods exist, 
from holding with a pipette to special, functionalized substrates. 

A common method is to place the sample onto a substrate of freshly cleaved 
mica, which has a charged, hydrophilic surface, to which proteins and other 
biomolecules bind easily. Sometimes, the mica is treated with a special solution 
such as APTES,’ which changes the surface charge. Additional coating of the mica 
with gelatin has proved to be effective when imaging bacteria parts [14]. Another 
practical method is a gold substrate coated with protein—reactive monolayers, 
making use of intermolecular binding forces for immobilization. 

More restrictive physical methods enclose the sample in a gel-like agar-agar or 
a porous medium, to avoid displacement. For instantaneous measurements, these 
methods are advantageous, supporting the natural height of the samples. However, 
they become problematic when the samples begin to proliferate, as cell cultures 
usually do. 


8.2.5 Cantilevers 


Cantilevers are characterized by their material, tip shape and mechanical para- 
meters such as spring constant, resonance frequency, and quality (Q) factor. 
For biological applications, cantilevers of silicon or silicon nitride are usually 
chosen. This material is chemically inert and can be doped to dissipate static 
charges. Also, the material allows a high Q factor for high sensitivity. When 
working in liquid, the pH value and electrolytes influence the interacting forces 
between tip and sample. For instance, under physiological conditions, silicon is 
charged negatively. 

Different treatments exist to allow a specific surface charge or hydration 
properties, namely plasma treatment or silanization. Silanization is also used for 
modifying the surface chemistry of substrates (for instance DNA binding to mica 
treated with APTES, Section 8.2.4). Tips can be functionalized with chemical 
coating (often as a preparative measure for biological coating) or biological 
coating, which can cover different interactions between receptor and ligand, 
molecules and cells, or cells and cells. Another option of cantilever modification is 
a reflective coating of the cantilever, which can be beneficial, since the biological 
buffer solution can dim the reflected laser beam. 

The tip shape is of crucial importance to the imaging quality. However, 
although a smaller tip radius yields more precise images, for imaging biological 
samples sometimes a duller (thus broader) tip is preferable, as it lessens the risk of 
damaging the sample. For elasticity measurements, besides the usual conical ones, 
spherical tips are also used, as they allow a smaller indentation depth and a better— 
defined contact area. Nevertheless, these tips usually measure an average elastic 


? Aminopropyltriethoxysilane, an amino—functional chemical compound. 
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modulus, in comparison to the higher lateral resolution by conical tips, which allow 
local measurements at different regions of the cell. 


8.2.6 Video Rate AFMs 


Common AFMs are limited to a rather slow scan rate in the minute range, allowing 
the measurement only of static or slowly developing processes. Several institutes 
are independently developing a high-speed AFM, trying to overcome the mecha- 
nical limitations to the scan speed. By optimization of the mechanical performance 
of stages and cantilever, scan rates of 10 s per image can be reached. 

However, just as for the mechanical limitations, resonant frequencies stemming 
from the utilized piezoelectric actuators could not be completely overcome and a 
new approach has, therefore, been tested. Instead of avoiding resonant frequencies 
by the use of lower driving frequencies, a microresonant scanner was used, which 
utilizes these frequencies. The cantilever was enhanced by passive means to 
maintain tip-sample interaction. In combination with dedicated high-speed 
electronics, a system has been built, which is able to image an area of 3 x 3 um 
with a resolution of 256 x 256 pixels at 15 frames per second. 

As the forces acting on the sample are much smaller, it is possible to depict 
even soft samples rapidly. The manufacturer claims that their system can be easily 
combined with existing commercial systems, upgrading them to video-rate 
capability, while preserving the standard AFM methods. [15]. 


8.2.7 Advantages and Disadvantages of AFM for Biohandling 


In comparison to other microscopes, the AFM has several advantages when hand- 
ling and imaging biological samples. It actually cherry—picks from the abilities of 
other microscopes, making it a very powerful characterization device. From the 
previous sections, the advantages of AFM—based biohandling can be summarized 
as follows: 


e A major advantage of the AFM is the variety of media in which it can be 
used. Biological probes can be examined under physiological conditions, 
and offer the capability to work with living specimens. 

e The possibility to use the AFM tip for nanomanipulation can be very 
useful. For example, higher—lying protein rings can be moved away, or the 
separation of chromosomes can be combined with the extraction of DNA. 
A cell membrane can be indented, or an ion channel can be activated by 
applying stress. 

e The possible resolution lies in the sub—nanometer range. 

e Three—dimensional topological information is offered. 

e No special structure-changing preparation is needed. Samples do not have 
to be dried or electrically conductive, as for instance in SEM measure- 
ments, where the probes have to be dehydrated and coated with a conduc- 
tive material, if they do not possess these characteristics naturally. 

e Different characteristics of a cell can be measured, such as elasticity, con- 
ductivity, adhesion, etc. 
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However, owing to the functioning principle of the AFM, there also are some 
disadvantages, limiting the possible measurements, which are summed up as 
follows: 


e No real three—dimensional force measurements can be conducted due to 
the construction of the AFM. While lateral forces (x-direction for 
convenience) can be measured, it is not possible to detect deviations in 
either the z— or y—directions separately. 

e Another drawback is the scanning speed, which is usually much too slow 
for real-time imaging. The newly available video—rate AFMs might help to 
solve this problem, but for now, their scanning area is very small (approx. 
3 um). 

e The possibilities of nanomanipulation can also be a drawback: soft mate- 
rials can be compressed, rendering height information valueless, be da- 
maged, or be dragged from their initial position. 


8.3 Biological Background 


A brief section on the biological background is inserted here to give an idea of the 
dimensions and forces in regard to biological samples. Some keywords are 
amplified to give better understanding of the application sections. The topics 
covered include the mechanical, chemical, and electrical characteristics of cells as 
well as a short description of bacteria, ion channels, and intermolecular binding 
forces, all of which can be found in this section. For more detailed information 
than in the highlights presented here, biological textbooks such as [16] are recom- 
mended. 


8.3.1 Characteristics of Cells 


Biological cells can be classified into two kingdoms: prokaryotes (bacteria) and 
eukaryotes (animal, fungi, plant, and other cells). They differ in structure and DNA 
storage. While eukaryotes have a nucleus and organelles, and can be both 
multicellular and single cellular, prokaryotes lack a nucleus or special organelles 
inside. Viruses do not belong to these two kingdoms, as they are not able to 
reproduce themselves, but need a cell host to do so. They are mainly capsules 
containing DNA information, which can dock to living cells and infect and alter 
them. 


8.3.1.1 Mechanical Characteristics 

The mechanical characteristics of eukaryotic (and some prokaryotic) cells are 
mainly defined by their cytoskeleton, which consists of actin—filaments, micro- 
tubules, and intermediate filaments. The cytoskeleton is responsible for cell moti- 
lity, cytokinesis, ° and the organization of the inner structure of cells. Moreover, it 


; Cytokinesis: Cell division. 
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has great influence on the elasticity of the cell membrane. Cell motility is of 
special importance, e.g,. in wound healing, phagocytosis,’ or the building of neuro- 
nal nets. Thus, cell mechanics and measuring cell elasticity have always been of 
interest and were examined by different means (cell poking, optical tweezers, efc.) 
even before the development of the AFM. The AFM has the advantage of a much 
higher lateral resolution in comparison to these methods. Certain parts of the 
cytoskeleton structure are of special importance in cell elasticity. For instance, 
elasticity changes due to actin—filament and microtubule degradation has been 
observed [17], where cells, after having been exposed to actin—degrading chemi- 
cals, showed a considerably increased elasticity. The cytoskeleton is enveloped by 
and attached to the cell membrane, which today is seen as a lipid bilayer with 
freely moving lipids and embedded proteins, as proposed in the fluid mosaic model 
by Singer and Nicolson, which is still used today, albeit with some modifications 
[18]. 


8.3.1.2 Electrical Characteristics 

Neurons and muscle cells mainly exhibit certain electrical characteristics, which 
are of importance in, e.g., neurological transmissions. The membranes of these 
cells are excitable and have a potential, which is regulated by the ion transfer in 
and out of the cell via certain ion channels. This so-called action potential is a 
rapid, self—progressing state of excitation of the membrane. It is caused by a 
depolarization of the membrane, which is followed by an opening of voltage— 
activated Na” ion channels. Opening these channels lets Na’ ions pass into the cell 
to change the potential of the membrane surrounding the ion channel from —70 mV 
to +50mV. Besides voltage—activated ion channels, other types exist. A brief but 
more general description of ion channels is given in Section 8.3.3. 

The fact that neurological activity is determined by electrical currents can be 
utilized in the artificial activation of these processes. Recent experiments attempted 
to stimulate cells by using single—wall nanotubes as an interface [19]. Carbon 
nanotubes and their electrical and mechanical characteristics are introduced in 
Chapter 7. Potential shifts can also be found for photoresponsive plant material, 
where they are the result of the light-induced production of indoleacetic acid, a 
growth hormone. This effect is used by the plant to grow towards the light. As the 
material is rather sensitive, touching the surface can result in erroneous signals 
because of stress, in wounding the cell, or other damage. For sensitive measure- 
ments, therefore, the non-contact method of Kelvin probe force microscopy is 
applied [20]. Another field of interest is the use of biological material merged with 
electronics for miniaturizing electrical circuits. For instance, in the case of DNA, it 
is not its electrical properties in a biological context that are of interest, but its 
being a building block for those circuits and for biosensors. Here, the conductivity 
measurements are usually conducted in contact mode. 





4 Enveloping of large mostly phatogenic particles by the cell membrane, followed by 
internalization into the cell. 
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8.3.1.3 Chemical Characteristics 

The chemical characteristics of a cell are found within several areas, as in the 
building of the membrane (peripheral proteins), with special ion channels, which 
are ligand—activated and in intermolecular binding forces. Intermolecular binding 
forces, as experienced in antibody—antigen bindings, can help to examine e.g. the 
composition of cells. Because of the specificity of the binding pairs, they can be 
identified easily, and thus are used e.g. as marker proteins in fluorescence micro- 
scopy and other measuring methods. The membrane potential can additionally be 
determined by chemical differences (for instance Ca” concentration). 


8.3.2 Escherichia Coli Bacterium 


Escherichia coli is one of the most common objects in cell biology. This bacterium 
lives in the lower intestines of warm-blooded animals and is responsible for the 
digestion of food. It is very well analysed and its complete genome has now been 
completely decoded. While usually Æ. coli is a necessary and friendly bacterium, 
certain strains of the E. coli family are hazardous to humans and are able to infect 
previously harmless strains, making them pathological. 

A common problem nowadays with bacteria is the resistance to certain 
antimicrobial drugs, for instance, antibiotics, as a resistant bacterium can spread 
its alterations by transferring its corrupted DNA to other bacteria. This is done by 
cell-to-cell contact; the process is called conjugation. For building the cell 
contact, E. coli has small "hairs" (actually plasma tubes) on its cell membrane, 
technically called fimbriaes, or pili for very short hairs. An examination of these 
hairs can give information about conjugation, and help to avoid resistance 
transfection. 

While the bacterium itself measures 1-2 m in width and 2—6 um in length (thus 
being well visible under an optical microscope), the fimbriaes are several um long 
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Figure 8.5. An SEM image of E. coli (Rocky Mountain Laboratories, NIAID, NIH) 
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but only of 3-10 nm in diameter, which excludes them from investigation with 
an optical microscope. Figure 8.5 shows an SEM image of the specimen. 


8.3.3 Ion Channels 


The cell membrane is considered as a lipid bilayer (about 5 nm thick) with em- 
bedded protein molecules. Some of these protein molecules function as gates in 
and out of the cell; the size of these proteins lies in the nm range (Figure 8.6). They 
are responsible for the absorption and dispensation of ions, resulting in their name 
ion channel. In this way, they change the membrane potential, which is used for 
signal transduction. 

Ion channels are activated by different means. Voltage activation by the mem- 
brane potential, as for Na™—channels, which are important for signal processing in 
neurons and muscle cells, has already been described in Section 8.3.1.2. Other 
channels are ligand—activated, for instance, the nicotinic acetylcholine receptors, 
which respond to the binding of nicotine. The stereocilia in the inner ear, which 
enable us to hear, contain ion channels that respond to mechanical stimuli (thus 
termed "mechanosensitive" or stress—activated) as produced by sound waves. And 
the photoreceptors in the eyes of vertebrates are sensitive to light; for the more 
sensitive rod cells’ even a single photon can result in a detectable electrical 
response. 

Ion channels are a favorite target in drug research, as they are involved in 
diverse processes which lead to rapid changes in a cell, such as in cardiac muscles, 
for t-cell activation, and insulin release, alongside some directly ion—channel— 
based diseases such as the shaker gene, the brugada syndrome, and some sorts of 
epilepsy. Some venoms, for example, work by "blocking" ion channels, thus 


O 





Figure 8.6. Scheme of a ligand—activated ion channel, embedded in the lipid bilayer 





> Rod cells are brightness sensitive, enabling black and white to be seen even in dim light, 
while the cone cells, mainly working in brighter light, are responsible for the perception of 
colors. 
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prohibiting signal transduction, which results in paralysis. Stress—activated ion 
channels are of special interest in AFM handling, because due to the flexibility of 
using an AFM as a nanomanipulator in direct contact to the sample, these channels 
can be activated by the cantilever tip (Section 8.4.3.1). 


8.3.4 Intermolecular Binding Forces 


For different processes, the key—lock principle (Figure 8.7) is found in micro- 
molecular biology. As mentioned, some ion channels are activated by ligands. 
They possess a specific receptor, to which the ligand can bind, thus opening the ion 
channel. 

Other important examples of intermolecular bindings are processes in the 
immune system of the body. Antigens and antibodies build similar bindings, thus 
deactivating viruses or bacterical toxins, or activating carrier and t-cells to take 
care of invading microorganisms or parasites. Studies of these binding forces can 
aid in the development of medicine, as it helps to understand the activation 
processes of immune cells. It may also be possible to block harmful cells or viruses 
from docking to healthy ones. The binding forces lie in a range of some pN, 
requiring very sensitive and noise-free measurement methods. 

Binding forces between proteins are also responsible for adhesion between 
cells. These adhesion forces in cell-to-cell contact play a major role in biological 
processes, for instance in the immune system, when white blood cells attach to 
infected cells (Figure 8.8). The trapping centers needed for attachment build up or 
decompose in seconds. This is an effect which is also of interest in self-assembly 
tasks, in biological manipulation, and in characterization setups [21]. 
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Figure 8.7. Key—Lock principle, antigen and antibody binding 
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Figure 8.8. A lymphocyte attaches to an infected cell, owing to adhesive domains 


8.4 AFM in Biology — State—of—the—art 


This section gives an overview of the state-of-the-art in AFM-based biological 
research. Its versatility, as well as an insight into the areas which benefit from the 
newly available measurement methods, is presented, ranging from purely imaging 
tasks, over the characterization of different physical, chemical, or electrical 
properties, to the use of the AFM as a manipulator as well as the cooperation with 
other end—effectors. 


8.4.1 Imaging 


A major field of AFM application is visualizing biological objects. Due to the 
possibility of measuring with nanoscale resolution in aqueous solution, this option 
is highly attractive for research and can give insight into molecular processes. The 
possibility of working in a buffer solution with non-static samples offers the 
possibility to visualize processes as well as taking mere snapshots. 

For example, the effect of poison on cell membranes has been examined by the 
CeNs group at the Ludwig—Maximilian University [22]. The poison contains an 
enzyme, which lets the cell membranes degrade. These enzymes dock to defects 
in the membrane, where they begin to react with the sample. The resulting rifts that 
gape in the membrane have a width of merely 10 nm, and holes of around 100 nm. 
Both can be imaged by the AFM. The knowledge of how enzymes work can be of 
benefit in drug development, for example, molecules similar to the imaged ones are 
found in the gastric acid. 

Other investigations at the Max Planck Institute for molecular cell biology and 
genetics have visualized the reaction of the proteins responsible for intercellular 
communications [23]. These gap junctions exchange molecules and signals. At the 
connexine, 26 gap junction surface pores are formed, which close under the 
influence of Ca”. 

Rare—earth metal ions are used as an additive in fertilizers for agriculture. To 
analyze the influence of the ions on living specimens, the reaction of E. coli 
bacteria exposed to La ions has been measured by means of AFM and SEM 
imaging [24]. The imaging was conducted in MAC mode, where only small forces 
act on the specimen. From the surface analysis of the bacteria, it could be seen 
that the surface roughness and the permeability had increased after a treatment with 
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La** ions. These are structural alterations, on which the influence of La’ in 
organisms is based. 


8.4.2 Physical, Electrical, and Chemical Properties 


8.4.2.1 Elasticity and Stiffness Measurements 

The elasticity of a cell membrane is an indicator of several changes — for example, 
older cardiac cells tend to get stiffer, while cancer—infected cells generally expose 
a softer membrane. Other findings are the change of the whole cytoskeleton, which 
influences the cell motility — whichis significant, e.g., when healing a wound in the 
skin. The specific conditions underlying these measurements have already been 
discussed. 

For instance, cell volume control is influenced by the cytoskeleton. Experi- 
ments tackling this area were conducted by [25], where the influence of osmotic 
stress on the stiffness of kidney cells has been investigated. Physical changes such 
as the detachment of the actin cortex from the membrane due to swelling of the cell 
could be indicated by the measurements of the Young's modulus. Different 
measurements with two types of probes have been conducted, one with a spherical 
tip for an average measurement on a cell, or conical tips for local measurements in 
the cell's center or the periphery, and imaging. The mechanical changes of the cells 
due to osmotic stress were proven to be reversible. It has been found that the 
swelling of the cell due to osmotic reaction gets ahead of the cytoskeleton—based 
physical adaptation of the cell, thus rupturing the actin cortex or detaching the 
cytoskeleton from the membrane. 

In the inner ear, mechanosensitive hair cells are responsible for converting 
sounds into electrical signals. The elastic properties of these hair cells are influ- 
enced by filament crosslinking between neighboring hairs and their cytoskeleton. 
The AFM allows unique experiments, where a single stereocilium® can be 
displaced. To ensure that the mechanical stimulus targeted at one hair is not 
stimulating neighboring hairs, the coupling of these hairs was investigated. The 
first experiment included the displacement of individual stereocilia with an AFM 
tip, in order to measure the stiffness of the stereocilia in relation to their position. 
The other experiment tested the force transmission to neighboring stereocilia. Up 
to two stereocilia were stimulated by an end-effector, while the whole hair bundle 
was scanned with an AFM probe. It was found that the influence of adjacent 
stereocilia was rather small, and the stimulation of a single stereocilium is, 
therefore, possible. This is a good basis for measuring the reaction of single 
transduction channels (which are mechanosensitive ion channels, Section 8.3.3), 
for which the exact knowledge of mechanical characteristics of the stereocilia is of 
primary importance [26]. 


Stereocilia are mechanosensing organelles of hair cells, working for instance as an acoustic 
sensor in the cochlea of the human inner ear. 
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8.4.2.2 Intermolecular Binding Forces 

Recognition force measurements for chemical analysis have, for example, been 
conducted by the University of Linz, where new methods for antigen—antibody 
binding force measurements were developed [27]. For these measurements, the 
tip of the cantilever was modified with a halved antibody, bound to it by a flexible 
interface. This modification ensured that binding with only one antibody at a time 
occurred. The cantilever can be used in dynamic mode, allowing it to take both 
topographical and recognition pictures at the same time. These methods were used 
in investigations concerning the localization of the well-studied single avidin- 
biotin interactions, to develop a stable setup for topographical and recognition 
imaging. 

Another field of interest for the analysis of intermolecular forces is the study of 
biosensors. There are a lot of different biosensors, all of which are based on the 
principle of an analyte binding to a biological interface (receptor). These binding 
forces are of interest in the development of new sensors. For instance, there are 
receptors which change their structure when a ligand (analyte) binds to them. 
Using AFM measurements, these rigidity changes can be analyzed, and the data 
can be used for the optimization of existing biosensors or the development of new 
ones. The are some systems which as a sensor use nucleic acid, which is more 
sensitive and specific in its function. However, the isolation of nucleic acid has 
proven to be very complex and these systems are still, therefore, very challenging 
[28]. 


8.4.2.3 Adhesion Forces 

For the measurement of cell adhesion, the AFM was utilized to advance experi- 
ments into the molecular range, where previous methods of adhesion measure- 
ments had failed to distinguish whether the bond was composed of many weak or a 
few strong bonds. At the Ludwig—Maximilian University, a setup for measuring 
cell adhesion at the molecular level has been built [29]. To provide a smooth 
surface for cell attachment, either the tip of a cantilever was removed, or a sphere— 
tip cantilever was used. Afterwards, the cantilevers were functionalized with 
adhesive molecules to let cells attach to them. For one experiment, cells were 
cultivated on the spherical tip, growing a cellular monolayer. The adhesion of these 
cells was tested with several different surfaces. Various parameters, such as contact 
time, were considered as well. For each surface, several force—distance curves were 
taken. Here, an increase of the adhesion load could be found, until the maximum 
adhesion force was reached. This is visible in the retraction part of the force— 
distance curve, indicated by the snap—back point (Section 8.2.2.4), followed by a 
series of detachment processes visualized by force steps. The setup was applicable 
to a multitude of different cells’ adhesions. 

More application—oriented measurements were conducted at Purdue University, 
where a material for small vascular grafts was researched to improve cell adhesion 
to this material [30]. The widespread atherosclerotic vascular disease is often 
treated with the implantation of synthetic vascular grafts, if the patient does not 
have sufficient vascular material of his own. In the course of five years, these 
synthetic grafts suffer from a strong decrease in potency of up to 95% in large— 
diameter, and 30% in small—diameter, vascular grafts. New materials are therefore 
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researched, to help improve vascular cell functions. The fibronection binding to 
differently patterned polylactic-co—glycolic acid surfaces was measured. Different 
nm-sized features where found to have a different effect on the binding of cells to 
the substrate. By means of topological AFM imaging, the cell absorption on the 
substrate could be measured. The findings are not only of interest in vascular 
treatment, as the researched binding protein fibronectin is of major importance in 
cell adhesion for many different types of cells. 


8.4.2.4 Cell Pressure 

AFM -based cell volume measurements have been accomplished by [31]. These 
measurements aided the experiments concerning the influence of the hormone 
aldosterone, which controls the water and electrolyte balance in the kidney, on 
cells of the cardiovascular system. Volume information was gained via the 
topological images, which contained height profiles of the cells. It could be shown 
that the volume of the cells actually increases after cultivation in a medium 
containing aldosterone. This finding can help in treating cardiovascular 
dysfunctions. 


8.4.2.5 Virus Shell Stability 

Force measurements on virus shells were conducted by [32], followed by a specific 
structure analysis. The capsids of viruses are not only involved in protecting the 
viral DNA, but also in selectively packing and injecting it to hosts. For measuring 
the structure and topological information, a series of jumping mode images were 
taken that allow the application of a controlled force to the capsid. The contact 
mode proved to be problematic — because of the lateral forces, the capsids were 
rolled over the substrate. After the "top" of a capsid was identified by topological 
imaging, a force—distance curve was conducted at this point. An inhomogeneous 
shell structure was found, as the calculated Young's moduli where closely arranged 
arround two different values. Information about resistance and stability was 
gained by repeatedly pushing the capsids. It was found that the capsids recovered 
their original height after a few ms. These experiments provided new information 
about shell structure, which is — because of its clever design — of interest in design 
techniques in nanotechnology, and proved that the capsids give not only chemical 
but also mechanical protection. 


8.4.2.6 Electrical Properties of DNA 

In the miniaturization of electronics, there are attempts to find suitable components 
for molecular devices. DNA, being well known and suitably sized, seems to be a 
good candidate for serving as a wire in nanoelectronics. As the electrical 
characteristics of such small-scale objects are expected to considerably differ from 
macroscopic objects, the exact measurement of these properties is of great 
importance. DNA conductivity has been a frequently discussed topic in recent 
years. Sample preparation and measuring techniques seem to have an 
overwhelming influence, since heavily differing transport properties touching the 
whole spectrum from conductor, semiconductor, supra conductor, to insulator, 
have been reported. Recent research confirms the conductivity of dsDNA under 
special conditions, while a self-assembled monolayer of ssDNA has been 


258 Saskia Hagemann 


discovered to serve as an insulator [33]. Here, different types of DNA were 
brought onto a gold substrate by coupling them with an thiol group. Some DNA 
strands were connected by means of another thiol group to a 10nm gold 
nanoparticle. This particle was contacted by the gold-coated AFM tip, and 
current-voltage curves were recorded. Other experiments include measurements 
without the gold nanoparticle. It has yet to be seen whether variations in these 
measurement techniques will come to a repeatable and usable result. 


8.4.3 Cooperation and Manipulation with an AFM 


8.4.3.1 Stimulation and Recording of Mechanosenstive Ion Channels 

The capability of the AFM tip to modify the sample at a nanometer level can be 
used to stimulate mechanosensitive ion channels. The exact knowledge of the 
mechanical properties of the channel—bearing structures are important. For 
instance, for displacing and characterizing single hair cells, the validity of AFM- 
based measurements has been clarified in [26]. The electrical response of one or 
more channels can be measured by an additional patch-clamp pipette [34]. Within 
the cochlea, individual stereocilia of hair cells were stimulated by displacement, 
analogously to the stiffness measurements described earlier. The electrical 
response, a transduction current, was simultaneously recorded via a patch-clamp 
pipette in a whole cell configuration. Even responses of single channels could be 
recorded, due to the ability to displace only a single stereocilium without 
significant influence on neigboring hair cells [35]. This method was not only 
shown to be useful in mechanical coupling, but also for the electrical response of 
the channels. These experiments can give a better understanding of the gating 
processes in the inner ear. 


8.4.3.2 Cutting and Extraction Processes on Chromosomes 

For the task of dissecting chromosomes, a sophisticated nanomanipulator was 
built, consisting of an AFM installed in parallel to a laser dissector [36]. The nano- 
manipulator not only allows interactive control via joysticks, but also haptic 
feedback, enabling the user to directly feel the applied forces. The abilities of both 
instruments to cut chromosomes were examined. Here, the cantilever tip was 
placed using optical control near to a spread of chromosomes. The chromosomes 
were then imaged, using the AFM, to find a suitable adjacent pair for dissection. 
The dissections were conducted by the user’s input via joysticks. It was discovered 
that the cuts accomplished by the cantilever were more precise when the cut width 
was smaller. While the minimal cut width of the laser dissector was found to be 
380 nm, the minimal cut width of a successful mechanical dissection was 280 nm. 
The force applied on the cantilever has to be chosen carefully, as insufficient force 
will result in a not fully dissected chromosome. Given an appropiate force, very 
precise cuts can be performed, e.g., the dissection of only one chromatide ("arm") 
of a chromosome. An additional benefit is the fact that after cutting the 
chromosome, DNA material adheres to the cantilever tip. The extracted DNA has 
been used in biochemical analysis, where the validity of gaining DNA samples 
using this method was approved. 
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8.4.4 Additional Cantilever 


For combining two independent AFM measurements, it would be very laborious to 
incorporate two custom AFMs. The use, preferably, of a self—sensing cantilever, 
mounted perhaps on a mobile platform, seems to be a viable alternative to an 
additional bulky custom AFM. In the framework of the MICRON project [37], 
such a preliminary platform has been developed, and first results proved to be 
promising. A robot bearing a self—sensing cantilever was able to image a compact 
disk surface. While the piezoelectric cantilever still has some problems reaching 
the accuracy of optical position measurements, some of these problems stem from 
electrical noise, which can be decreased by a clever design of circuits. Although 
the main idea of this project was a swarm of small, independent, cooperating 
robots carrying different tools (force sensor, pipette), further developments of 
AFM probes cooperating with a tool-carrying robot can be imagined. The 
advantage of such an independent second AFM cantilever could be the ability to 
use one of the cantilevers as a stress generator and the other as a sensor, in order to, 
e.g., experiment on stress transmission of cells. 


8.5 AMNS for Cell Handling 


The AMNS concept introduced in Chapter 1 can be utilized for handling and 
characterizing biological cells. Analogously to the AMNS presented in Section 7.5, 
a setup for mechanical cell characterization, initially for measuring cell 
elasticity, has been developed. As an optical sensor, an inverted optical microscope 
is used instead of an SEM, as the samples used are of a size still visible with an 
optical microscope, and measurements with living cells are proposed. 
Nevertheless, operating this station, e.g., in an ESEM (environmental SEM) is 
definitely possible. The force measurements are conducted with a piezoresistive 
cantilever. Similar to the experiments for CNT characterization, coarse positioning 
of the cantilever is conducted by the nanomanipulator, while fine positioning and 
the force—distance curves are carried out using the nanopositioning stages. 


8.5.1 Experimental Setup 


First measurements were taken using the setup depicted in Figure 8.9. The AMNS 
differed from other stations, as changes had to be made to adapt the station to the 
use with an inverted optical microscope and the handling of biological samples, 
which were available in containers different to, e.g., the CNTs. 

For this, the three degree—of—freedom piezo—scanning stage [38] was equipped 
with an attached holder for a glass slide or Petri dish containing the biological 
sample to be analyzed. The holder had to have a recess for the optical path and was 
positioned sideways to the piezo—scanning stage for the same reason. A modified 
base plate was mounted on the specimen stage of an inverted optical microscope 
[39], which was used as an optical sensor. The integrated capacitive sensors of the 
piezo stage allowed it to be operated in closed-loop mode. 
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Figure 8.9. Setup of the nanohandling station for characterizing biological cells 


The three—axes nanomanipulator [40] was attached to the base of the scanning 
stage. It offers a theoretical resolution of 5 nm in the x— and y-directions and 
0.5nm in the z-direction. As an end-effector for the nanomanipulator, 
piezoresistive cantilevers [41] were used. These AFM probes have a pyramid tip 
with a radius below 10 nm and a height above 17 um. 


8.5.2 Control System 


Figure 8.10 shows the control scheme used for the elasticity measurements 
conducted with the AMNS. The characterization of biological cells is performed by 
bringing the piezoresistive AFM probe in contact with the sample. The resulting 
bending of the cantilever due to interacting forces leads to a varying resistance, 
which is converted to a detectable voltage by a four—wire wheatstone bridge. The 
voltage signal is amplified and digitized by a custom—made bridge amplifier and 
transferred to the control PC via the universal serial bus. Different input possi- 
bilities are given. The user has the opportunity to interact with the system to 
manually perform the first experiments, beginning with the nanomanipulator 
positioning, which can be operated manually, by a joypad input device, or by the 
graphical user interface (GUI). Meanwhile, the image acquired from the optical 
microscope is giving visual feedback to the user. In addition to the height 
information gained from the focus scan of the microscope, the sensor data received 
from the piezoresistive cantilever are used as a safety mechanism, avoiding 
possible tip crashes. As a next step, the piezo—scanning stage holding the sample 
can be manually positioned to a desired location. Again, the force data gained from 
the piezoresistive cantilever are used to avoid a collision between the AFM tip and 
the sample surface. 
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Figure 8.10. Control scheme of the AMNS for cell handling 


The stage controller delivers an analog signal encoding the stage's exact 
displacement, so the stage can be operated in closed—loop mode. A haptic interface 
can be used as an auxiliary input method to move the scanning stage for fine 
positioning of the sample in respect to the cantilever. The forces acting on the 
AFM cantilever are rendered on the haptic device to give the user additional feed- 
back of his manual operation. 

To accomplish the highly automated measurements, a complex control system 
is needed. The system has to cover object recognition and tracking tasks, analyzing 
images gained by the inverted optical microscope. The use of an optical 
microscope as a visual sensor to recognize the three—dimensional position of end- 
effectors and biological objects has already been realized by a previously 
developed cell handling station [42]. This facilitates the application of path 
planning algorithms to avoid collisions during the performance of high level tasks. 
A client/server architecture has been implemented, allowing maximum possible 
flexibility. 


8.5.3 Calculation of the Young's Modulus 


This AMNS is designed for the mechanical characterization of biological cells, 
focusing on measuring the Young's modulus of the specimen. Before the 
measurements can be conducted, however, the cantilever must be calibrated. This 
calibration takes place on a hard substrate, similar to Section 7.5.3 and [43], and 
thus is not influenced by the soft specimen. For measurements concerning the 
specimen, owing to the softness of the material, special care has to be taken 
regarding indentation. Indentation results in force—distance curves considerably 
different from those taken on a hard substrate (Figure 8.2), and as mentioned there, 
the measured deflection of the cantilever is superposed with the indentation into 
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the material. The formulae and the theoretical background describing the 
calculation of the Young's modulus for soft specimens was given in Section 
8.2.3.1. 


8.5.4 Experimental Results 


First experiments were conducted in ambient air [44]. Dried tumor cells were 
chosen as specimens, which are adherent to the bottom of a Petri dish, forming a 
layer of cells and cell debris with a miscellaneous Young's modulus. The cells 
themselves have a diameter between 20 and 30 um, but are densely embedded in 
the cell debris. 

At different areas of the sample, force—distance curves were taken. The 
measurements indicate changing adhesion and elasticity, which is to be expected 
from the inhomogeneous layer. Mean values of a succession of ten force—distance 
curves were taken for evaluation, as depicted in Figure 8.11. Most areas show the 
characteristic behavior of softer material, the curves being slightly non-linear and 
having a smaller slope than theoretical curves for a hard substrate (cf Figure 8.2). 
Additionally, large adhesion forces can be found, as, on the retraction of the 
cantilever, the cantilever sticks to the sample for some time until it finally breaks 
free. The force—distance curves show the characteristic steps in the retraction part 
of the curve for contaminated or coated samples in ambient air, indicating different 
layers successively detaching from the tip, which seems reasonable for the cell- 
debris—coated surface (Section 8.2.2.4). The measured Young's moduli lie between 
577 kPa and 1.03 MPa, thus being close to gelatin, and seem reasonable for the 
gel-like biological material. Young's moduli of living cells are considerably 
smaller (50 kPa), while, e.g., bone is much harder (30 GPa). 

With the realization of a cell elasticity measurement station, a reasonable foun- 
dation for a versatile cell characterization station has been built. Further experi- 
ments are necessary to establish whether the required resolution and spring cons- 
tants for measuring softer and more delicate specimens can be achieved by the 
piezoresistive cantilevers. 
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Figure 8.11. Force distance curve on cell debris 
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8.6 Conclusions 
8.6.1 Summary 


The versatility and effectiveness of AFM-—based biocharacterization has been 
shown in the previous sections. Novel experiments, which were only made 
possible due to the unique abilities of the AFM, could be conducted, helping to 
understand basic cytological processes. Not only has the possibility to visualize 
processes on a cellular level been shown, for example, the destructive influence of 
enzymes on the cell membrane, but also has the difficult analysis of the electric 
behavior of DNA, the analysis of intermolecular forces in the pN—regime, and the 
transduction processes of sound into electrical currents in the inner ear. However, 
many processes in the micro— and nanoworld are not fully resolved, for example, 
the mechanics of cell motility or cell-to-cell contact in the conjugation of 
bacteria. More sophisticated setups can help to investigate these phenomena, 
expanding, e.g., pharmaceutical possibilites. 

Despite all of its versatility, the AFM has some major problems, mostly due to 
construction issues, which hamper the development of automated bio- 
nanohandling stations: 


e No real-time imaging, due to low scan-rates; 

e No three—dimensional force measurements, due to the nature of the laser— 
detection system with a two- or four—segment diode; 

e Manipulation and imaging require different types of tips for optimal 
results. 


A solution to these problems has to be found, either by changing the underlying 
constructional issues, or by enhancing AFMs, e.g., by the use of additional sensors. 


8.6.2 Outlook 


For some of the problems listed above, solutions are currently being developed. 
For other issues, methods of resolution are proposed in the following. 

For the real-time imaging problem, a solution might have arrived with the new 
video—rate AFMs, of which some are already commercially available systems. 
However, the scan range of these systems is very limited, confining the measure- 
ments to a 3 x 3 um area, which is not satisfactory for most characterization and 
imaging tasks. Another option, which can compensate for the lack of real-time 
imaging is the possibility of including a haptic interface, which gives force feed- 
back to the user while the AFM is still updating the image [45, 46]. Generally, the 
enhancement of the cantilever by piezoresistive or electrical sensing methods can 
solve the problems in measuring three—dimensional forces, as distinct forces 
beyond lateral ones can be registered separately. A combination of laser and piezo— 
based cantilever position measurements could be a meaningful enhancement to 
force measurements. However, the noise problems and the accuracy of piezo— 
based self—sensing options are unresolved issues. 
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The problem of needing two different types of cantilevers or AFM modes for 
optimal imaging and measuring or manipulating has become obvious from the 
previous applications. For example, in the case of measuring the virus capsid 
stability, a contact cantilever was necessary for the force measurements, but a slow 
imaging technique (jumping mode) had to be chosen, as normal contact mode 
applied too strong lateral forces. For measurements of the cell elasticity, a 
cantilever with a spherical tip is sometimes chosen for several reasons, but for 
imaging, this cantilever is not suitable and has to be exchanged. Other problems 
occur with the contamination of the AFM probe after, for instance, conducting 
force—distance—curves or cutting, which afterwards degrades the imaging quality. 
Overall as an ideal choice, imaging and manipulating usually call for different 
probes. A solution for this problem can be derived from the idea which has already 
been implemented for all of the previously described AMNS, where the visual 
sensor (either SEM or light microscope) and manipulator are decoupled. 

Analogously, a commercially available AFM is proposed to be only used for 
the task of imaging samples, while an additional end-effector is used for 
manipulation and measurements. This setup could have another advantage: the 
second end-effector is not restricted to being a cantilever, but could be a mani- 
pulator of any kind, such as an injection pipette or the like. This way, injection, 
patch-clamp, as well as cutting or force measurements could be conducted. Several 
different tools could be utilized. The endeffector should ideally be self—sensing, 
which is a large development area, e.g., including three—dimensional force- 
sensitive pipette holders. The requirements for the necessary control system are 
very high, as different spatial ranges from micro— to nanometers have to be 
covered, and visual control of the end-effector by an optical microscope and AFM 
control of the manipulation effects in the nm—range have to be achieved. 
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9.1 Instrumented Indentation 


Instrumented indentation is one of the most commonly used methods to determine 
the mechanical properties of materials. This method is based on the penetration of 
a body with a known geometry into the material’s surface. Both the force (or load) 
necessary for this penetration and the depth of indentation have to be measured, 
either separately or simultaneously. 

Measuring these two quantities simultaneously leads to load—depth—curves. 
Applying further analysis to these load—depth—curves finally leads to the basic 
material’s hardness and Young’s modulus of elasticity. Not only bulk materials are 
under test with this method, but also coating systems consisting of two or more 
layers of different materials, e.g., a thin film on a substrate. 

While the usage of indentation tests on the macroscale has been known for 
decades, the method of application in the micro- or even nanoscale has been 
developed only recently. In the first part of this chapter, the basics of instrumented 
indentation and its analysis are given. The second part deals with microrobot-based 
nanoindentation. Some first experiments to characterize an electrically conductive 
adhesive are presented here. 


9.1.1 Sharp Indentation 


9.1.1.1 Introduction 

The instrumented indentation method has been derived from the classical hardness 
testing method. Hardness has been defined as the resistance of a material against 
the penetration of a hard body with a known geometry, called an indenter. After 
the penetration, the residual imprint is measured and used in different relations to 
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calculate the hardness. Only the plastic deformation of the tested material is taken 
into consideration [1]. For example, the Martens hardness HM can be calculated by 


[1]: 


HM = a = -eaS for a Vickers indenter, (9.1) 
A, 26.43-h 
F F ARE 

HM = — = ———— for a Berkovich indenter, (9.2) 
A, 26.44-h? 


where F is the applied force, A; is the area of the imprint remaining after the 
indentation test, and A is the remaining indentation depth. 

The principal difference between the classical hardness method and the instru- 
mented indentation method is that during an instrumented indentation test, the 
penetration depth and the indentation force are measured continuously. It is, in 
practice, an evolution of the contact response during the loading and unloading 
stages of the experiment [2]. From this test, the elastic and plastic deformation of 
the tested material can also be determined. The test provides load vs. indentation 
depth curves, which give helpful information about the mechanical properties of 
materials. Figure 9.1 shows such a typical curve. From this curve, additionally to 
the maximum indentation depth and force, the unloading contact stiffness S and the 
contact depth h. can be determined. With these parameters, using the theory of 
Oliver and Pharr [3], the Young’s modulus and the hardness of the tested material 
can be calculated. This procedure will be explained later in Section 9.1.1.5. The 
method can also be applied for coating systems, but there are some difficulties 
especially with regard to very thin coatings. In this case, in order to avoid the 
influence of the substrate material, special testing conditions should be considered. 
This will also be discussed in Section 9.1.1.5. 

The method is called sharp instrumented indentation because of the shape of 
the indenter, which can be conical for Rockwell indenters or pyramidal for 
Vickers and Berkovich indenters [4] (Figure 9.2). 

This test method has some advantages: in principle, it can be applied on 
specimens and components of any geometry, requiring only a small quantity of 
material. There are also several disadvantages in evaluating the test results from 
sharp indentation experiments. The extent to which these techniques can be used 
to quantify material properties is limited by the current understanding of the 
complicated material response during indentation experiments. For example, 
obtaining constitutive equations from indentation techniques has traditionally been 
limited to stress-free, perfectly elastic plastic material, not, therefore, giving critical 
information regarding residual stress and strain hardening. 
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Figure 9.1. Diagram of load vs. indentation depth with the principal parameters needed for 
calculating the Young’s modulus and the hardness of the tested material 
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Figure 9.2. Sharp indenters: a. Rockwell indenter; b. Vickers indenter; c. Berkovich 
indenter (from [4], courtesy of Synton-MDP AG, Switzerland) 
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9.1.1.2 Basic Concepts of Materials Mechanics 

The basic concepts of the mechanics of materials can be explained by looking at a 
cylindrical bar under axial forces [5]. An axial force is oriented parallel to the axis 
of the structural element. 




















Figure 9.3. Schematic representation of a bar under axial load: a. initial length of the bar; 
b. bar elongated under an axial load 


If the bar with an initial length L is loaded under the axial load P, it will be 
stretched, so the length will be L + ô, where ô is the elongation of the bar. 
Furthermore, if a cross-sectional area A is considered, the stress ø in the bar can be 
defined by [5]: 


P 
o=—, 9.3 
F (9.3) 


£=— (9.4) 


Hook’s Law expresses the linear relationship between stress and strain [5]: 
o=E-ée, (9.5) 


where Æ is the modulus of elasticity of the material. It is also called Young’s 
modulus. 


9.1.1.3 Similarity Between Sharp Indenters of Different Shape 
Finite-element calculations have shown many similarities between sharp indenters 


[2]: 


there is a similarity in the stress field within the fully plastic regime; 
there is a similarity in the obtained hardness value, for a ratio between 
Young’s modulus E and the uniaxial stress ø, at a totally characteristic 
strain £, of 0.1, E/a ,> 150; 

e there is a similarity in the surface deformation after an indentation test with 
a sharp indenter; the average value around the imprint after pyramidal 
indentation is the same as after indentation with a conical indenter. 
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9.1.1.4 Indentation Ranges: Nano-, Micro-, and Macroindentation 
The indentation tests can be classified as a function of the magnitude of applied 
load in nano-, micro-, and macroindentation (Figure 9.4). It is called: 


e nanoindentation, for the maximum load applied being up to 0.5 N; 
e microindentation, for the maximum load applied ranging from 50-100 N; 
e macroindentation, for the maximum load applied being more than 100 N. 


Corresponding to these load ranges, there are differences in respect of the response 
behavior of the indented material [6]: 


e in the nanoindentation range, it is a response due to discrete phenomena, 
such as dislocation glide; 

e in the microindentation range, it is a mesoscopic behavior, a response of 
single phases and microstructural units; 

e in the macroindentation range, it is a macrosopic behavior, a response of 
the bulk material. 









Pi 
macro 
(Pra > 100 N) 
micro 
(Paa ~ 50-100 N) 
nano 
(Pan ~ 0.5 N) 
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Figure 9.4. Indentation in nano, micro, and macro ranges 


9.1.1.5 Analysis of Load—Depth Curves 

The first researcher to use the instrumented indentation (with a spherical indenter) 
was Tabor as early as 1948 [3]. Later on, in 1961, Stillwell and Tabor [3] used a 
conical indenter. They tested metals and learned from their results that the 
unloading stage of load—depth indentation curves can be used to calculate an elastic 
modulus. In 1970, Bulychev et al. [3] also used instrumented indentation and 
further analyzed the unloading stage of load—depth indentation curves. A typical 
curve is shown schematically in Figure 9.5. They defined for the first time the 
stiffness S of the upper portion of the indentation test’s unloading stage [3]: 


=a if 
dh Jr (9.6) 
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in which S is the stiffness of the upper portion of the indentation test’s unloading 
stage, P is the indentation load, H is the indentation depth, E, is the reduced 
modulus, defined below in the Equation 9.7, and A is the contact area between the 
tip of the indenter and the surface of the tested material under indentation load. 
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Figure 9.5. Example of a load—depth indentation curve 





i. (9.7) 


where E is the Young’s modulus of the tested material, v is the Poisson’s ratio of 
the tested material, Æ; is the Young’s modulus of the indenter material, and v; is the 
Poisson’s ratio of the indenter material. 

Because determining the contact area A is difficult, in 1983, Pethica, Oliver, 
and Hutchins [3] defined the indenter area function, also called the shape 
function. The latter is the cross-sectional area of the indenter as a function of the 
distance from its tip [3]. First of all, it should be noted that at the maximum 
indentation load Pmax, the surface of the indented material will have practically the 
same shape as the external shape of the indenter at the correspondent depth, the 
maximum indentation depth Anax- Consequently, if this maximum indentation depth 
is known from the experimentally recorded load—depth curve, it is easier to 
determine the contact area from such an indenter area function. 
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In 1986, Doerner and Nix [3] further developed this method based on their 
experimental observation that, in the upper portion of the unloading stage of the 
indentation test, the elastic behavior of the indentation contact is like that of a flat 
cylindrical punch. They assumed further that the unloading stage of the indentation 
curve can be approximated by a linear portion, and they determined the indenter 
area function by using the extrapolated depth of the “linear” unloading stage 
corresponding to zero loads. If the contact area is known, the Young’s modulus can 
be calculated from Equation 9.6. They proposed to calculate the hardness from 
Equation 9.8 [3]: 





Hes ome (9.8) 


where Pmax is the maximum indentation load and A is the contact area at the 
maximum indentation load. 

In 1992, indentation tests by Oliver and Pharr [3] suggested that the assumption 
of Doerner and Nix was too inaccurate and they further contributed to the 
development of this test method. In contradiction to Doerner and Nix, they took 
into consideration the curvature of the unloading stage of load—depth indentation 
curves. Additionally, they provided a new method for determining the depth, with 
which the area of contact should be calculated. 

The necessary parameters for calculating the Young’s modulus and the 
hardness can be identified by this method. In Figure 9.6, a cross-section of an 
indentation is shown. 
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Figure 9.6. Diagram of a cross-section of an indentation 
From this figure, it can be seen that the total indention depth / is [3]: 


h = h, F h, 2 (9.9) 
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where h. is the contact depth, i.e., the depth at which there is contact between 
indenter and the material surface, and h, is the displacement of the indented surface 
at the outside edge of the contact between indenter and material. 

After Oliver and Pharr [3], the necessary parameters for calculating the 
Young’s modulus and the hardness of the indented material are the maximum 
indentation load Pmax, the maximum indentation depth Amax, and the initial unload- 
ing contact stiffness S. S is only measured at the maximum indentation load. They 
are depicted above in Figure 9.1. The most important relationship is: 


E Lane 
r 2 ne 


which is Equation 9.6, but rewritten in such a way that the reduced modulus £, 
can be calculated. 

One of the most difficult problems is to determine the contact area at the 
maximum load A. Oliver, Hutchings, and Pethica assumed in 1986 [3] that 4 can 
be written as an area function F(h), which takes into consideration the cross- 
sectional area of the indenter and the distance from the tip: 


(9.10) 


A=F(h,). (9.11) 


This function F must be determined experimentally. Oliver and Pharr [3] proposed 
the following relationship for calculating the contact depth A.: 





2 (9.12) 


where € is a constant which depends on the geometry of the indenter: ¢ = 0.72 for 
a conical indenter, ¢ = 0.75 for a paraboloid of revolution, and ¢ =/ for a flat 
punch [7]. 

In 2004, Oliver and Pharr [7] proposed a slightly modified relationship for 
calculating the unloading stiffness S: 


pe as, (9.13) 
mT 


F 


If we compare this relationship with Equation 9.6, it can be seen that £, which is a 
correction factor, was considered as unity. This is true if it is assumed that the 
material is perfectly elastic and that there are only small deformations. Real 
indentation experiments have shown that the indenters used are non-axisymmetric 
and that large strains occur. Over the years, many researchers have performed 
numerical calculations and finite-element analysis for exactly calculating the 
correction factor. In 1995, Hendricks found a value of 6 = 1.0055 for a Vickers 
indenter and a value of $ = 1.0226 for a Berkovich indenter. 
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Oliver and Pharr also presented in [7] the changes made as a result of research 
in order to improve the methodology and to better understand the mechanisms 
involved in an indentation test. For example, they investigated an error source in 
calculating hardness and Young’s modulus, e.g., the pile-up in the case of conical 
and Berkovich indenters. If during the indentation such a pile-up occurs, the 
contact area between the indenter and the surface of the tested material is 
significantly larger. This leads to relevant errors, e.g., up to 50% overestimation of 
the Young’s modulus. The material properties that affect the pile-up are the ratio of 
the reduced modulus to the yield stress E,/o, and the work-hardening. They have 
found a parameter which can help to estimate the indentation behavior of materials. 
It is the ratio of the final indentation depth to the depth of the indentation at 
maximum load, Ahmas It is possible to calculate this parameter from the 
experimentally recorded indentation curves. Additionally, it has been found that, if 
this ratio is A/a, > 0.7, the indented area should be analyzed and measured in 
order to avoid underestimations of the Young’s modulus. 

Cheng et al. [8] developed a method that can correct the influence of the pile- 
up without measuring the true contact area. Analyzing indentation curves, they 
have found that the ratio of the irreversible work to the total work, 
(Wior— Wa) Won is independent of the work-hardening of the indented material. 
Here, Wor is the total work of indentation, which is equal to the area under the 
loading curve, and W, is the work recovered during unloading (Figure 9.7). 
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Figure 9.7. Irreversible work W, of indentation from an indentation curve 


One of the most important improvements of this method was the use of the 
“continuous stiffness measurement” technique [7], which means that during the 
loading stage of the indentation test, the stiffness is measured continuously; 
additionally, a frequency-specific amplifier is used for imposing a small dynamic 
oscillation on the recorded signals of force and indentation depth and for 
measuring the corresponding amplitude and phase of these signals. This was only 
possible through the advance achieved in measuring techniques over the years. 
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Based on the “continuous stiffness measurement” technique, Oliver and Pharr 
[7] improved the calibration procedures for the stiffness of the testing machine and 
for the indenter area function. Using this method, it is no longer necessary to 
perform multiple indentation tests on different reference materials for calibrations, 
and consequently, the necessary time for the calibrations is significantly reduced. 

Myiake et al. [9] analyzed the influence of the pile-up on the results obtained 
by nanoindentation experiments. They conducted such experiments by means of a 
commercial AFM (atomic force microscopy) system, equipped with a rectangular 
stainless steel AFM cantilever with a diamond tip, on reference specimens (fused 
silica and single-crystal silicon). For the determination of the contact area, they 
analyzed the data by means of the Oliver and Pharr method and by direct 
measurement of this area with an AFM. Figure 9.8 shows a schematic 
representation of a cross-section of an indentation with pile-up. 
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Figure 9.8. Schematic representation of a cross-section of an indentation pile-up (from [9], 
with kind permission of the Japanese Journal of Applied Physics, Institute of Pure and 
Applied Physics) 


They found that the results obtained by the Oliver and Pharr procedure were 
overestimated. Therefore, it can be concluded that for very precise results, it is very 
useful to measure the contact area of the residual imprint after the indentation tests. 
As we have seen, the only difference in calculating the Young’s modulus by 
instrumented indentation for indenters with a different shape is the factor B 
introduced by Oliver and Pharr [7] in Equation 9.13 for calculating the unloading 
stiffness S: 6 = 1.0055 for a Vickers indenter and 6 = 1.0226 for a Berkovich 
indenter. Furthermore, the relationship for calculating the reduced modulus Æ, can 
be written as: 


vr 5 (9.14) 


Franco et al. [10, 11] developed this method in order to calculate the Vickers 
hardness and to obtain the Vickers number HV: 


HV = — , (9.15) 
Vickers 
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where A yickes 1s the contact at the maximum load for the Vickers indentation. They 
used the following relationship for calculating this area: 


d? 
ye ees (9.16 
Vickers 1 854368 ) 


where d is the average diagonal of the Vickers indenter, which can be calculated 


by: 
S fz 
d=f-—.,|—, 9.17 
B T (9.17) 


where 2, S, and E, are parameters from instrumented indentation tests. 

All of these considerations about the method of Oliver and Pharr for 
determining the mechanical properties of materials are correct for bulk materials. 
For coating systems, the method can also be used, but the influence of the substrate 
material should be considered [12]. Therefore, for determining the properties of a 
coating, the indentation depth should not exceed 1/10 of the total thickness of the 
coating. 


9.1.1.6 Applications of the Sharp Instrumented Indentation 

Sharp instrumented indentation is a technique widely used for determining the 
mechanical properties of materials ranging from metals and ceramics [2, 12-14], 
through polymers [15], to biological materials [16]. 

Zhao et al. [12, 17] determined the uniaxial residual stress and the mechanical 
properties of thermal barrier coatings (TBC) for aerospace applications by 
instrumented indentation. The determination of the residual stresses in a coating 
system is very important, because these stresses affect the integrity and reliability 
of such material systems. Moreover, the presence of the residual stresses (Figure 
9.9) in a material system affects the contact area of indentation. 


Z 





Figure 9.9. Schematic representation of a material with compressive residual stresses 
(reprinted from [17], with permission from Elsevier) 
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For example, if the residual stresses are compressive, the contact area will be 
smaller than in the case of a material system without residual stresses. The results 
have been verified by means of finite-element analysis. 

Mircea et al. [13] investigated TBC systems that comprised a single crystal 
super alloy substrate, a plasma-sprayed metallic (NiCoCrAlY) layer with a 
thickness of 180 um, and a ceramic coating with a thickness of 280 um, applied by 
electron beam physical vapor deposition. The test method of investigation was the 
Rockwell indentation. Rockwell indentation tests generated delamination cracks 
of a butterfly shape, while the delaminated coating did not buckle. Such fracture 
behavior was in contrast to results reported earlier, which show circular 
delamination cracks and buckling [18]. In order to understand the specific fracture 
behavior, displacement-controlled indentation tests were conducted on the TBC 
system and on the metallic substrate, with and without metallic coating. The shape 
of the imprints after indentation and the fractured coatings were investigated by 
means of optical and scanning electron microscopy. It was found that the observed 
butterfly shape of the delamination was caused by anisotropy in the deformation 
behavior of the substrate material. Because the coating was thick compared to the 
indentation depth, the interaction indenter tip/coating affected the available energy 
for propagating delamination cracks. For example, after heat treatment, the energy 
for achieving a certain indentation depth increased, and shorter delamination 
cracks were observed. In some cases, it was possible to attribute the formation of 
shear cracks in the ceramic to specific portions of dissipated energy achieved from 
the load—displacement curves. However, during the penetration of the ceramic 
coating, many energy-dissipating processes are acting simultaneously. Thus, it is 
generally not possible to attribute certain amounts of energy to only one 
mechanism. For the case of thick coatings, it is proposed to modify the classical 
Rockwell indentation test by removing the ceramic directly underneath the 
indenter, in order to achieve these conditions. In this way, the computation of the 
energy available for propagating the delamination crack can be made. 

Pharr [19] has shown that by nanoindentation it is possible to determine the 
fracture toughness of brittle materials (e.g., ceramics) by measuring the radial 
cracks that occur after indentation and by using a simple relationship. The 
relationship for the calculation of the fracture toughness K, is [19]: 


5 
K. =a f£) {SI (9.18) 


C 


where P is the maximum indentation load, a is an empirical constant depending on 
the geometry of the indenter, and c is the length of the crack. 

Other researchers [15] have mentioned the use of instrumented indentation for 
characterizing polymeric materials. These materials have mechanical properties 
between elastic solids and viscous fluids. They have shown a change in displace- 
ment at a constant force (creep). Polymeric materials are more compliant in 
comparison to metals and ceramics, which are normally characterized by instru- 
mented indentation. 
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Schéberl et al. [20] investigated the mechanical properties of human teeth by 
instrumented nanoindentation. The teeth investigated were tested just after 
extraction, in order to determine their hardness and elastic modulus under 
conditions very close to in vitro. They also determined the mechanical properties 
of teeth after about 15 minutes; in this time, the containing water was expected to 
evaporate. It was found that the mechanical properties of teeth depend on the water 
content. 


9.1.2 Spherical Indentation 


The name spherical indentation refers to the shape of the indenter, which is a 
sphere (Figure 9.10). Spherical indentation began with Hertz’s theory in 1880. 





=< spherical 
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Figure 9.10. Device with a spherical indenter 


9.1.2.1 Comparing Spherical and Sharp Instrumented Indentation 

Spherical indentation has some advantages over sharp indentation. First of all, it 
has a variable contact pressure, which is suitable for studying pure elastic contact. 
Moreover, it has a variable contact curvature radius (Figure 9.11). Therefore, 
this method shows the ability to simulate real contact conditions. However, it is 
difficult to measure the contact radius experimentally, especially in the elastic 
regime that is characteristic for nanoindentation. 

While sharp indentation depends on the indentation depth, spherical indentation 
depends on the sphere radius [21], the hardness increasing with a decreasing 
indenter radius. 

In the case of spherical indentation, the roughness of the surface of the tested 
material is important; an asperity can lead to a false evaluation of the contact area 
at the maximum load [22]. It is also very difficult to manufacture indenters with a 
perfect spherical shape and with a required radius. Swadener and Pharr [22] 
developed a calibration method for spherical instrumented indentation, by perform- 
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Figure 9.11. Schematic representation of the variable contact curvature radius 


ing indentation tests on two different ceramic materials at the same indentation 
depth. Using this method, they were able to determine the real indenter radius 
and the machine compliance. 


9.1.2.2 Analysis of Load—Depth Curves Using Spherical Indenters 

Because of the different shape of the indenters, the relationship between load and 
displacement is also different [7, 23]. The contact depth for the spherical indenter 
is determined by applying Hertz’s theory for a spherical indenter with a radius R, 
that penetrates a spherical hole with a radius R,, which finally corresponds to the 
remaining impression in the surface of the tested material after the indentation test. 
The equivalent radius R may be defined by: 


URS (9.19) 
R R, 


w|- 


The load-displacement relationship for the case of the spherical indentation is 
described by: 


P=. JRE, (h-r). (9.20) 


All of the parameters in Equation 9.20 have already been introduced in Section 
9.1.1.5. The unloading stiffness S for the spherical indentation can be written as: 


S=2-VR-E,-(h-a, (9.21) 
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Taking h = Ama for evaluating the expression above for the maximum indentation 
depth, and taking into consideration that the parameter ¢ (Section 9.1.1.5) for the 
case of spherical indentation is 0.75, a contact depth /, follows: 


p = ame (9.22) 


which is characteristic for spherical indentation. 

The considerations made in Section 9.1.2.4 for the determination of the 
mechanical properties of coating systems are also valid for the case of spherical 
indentation. 


9.1.2.3 Applications of Spherical Instrumented Indentation 
The applications of spherical instrumented indentation are similar to those of sharp 
instrumented indentation. There are only few differences in the application ranges. 
One of them is the application of spherical instrumented indentation to determine 
stress—strain curves of bulk materials and coating systems. These curves are useful 
for determining the transition from elastic to a plastic behavior of a material [24]. 
An important application field of spherical instrumented indentation is the 
study of the damage behavior of layered structures under concentrated load (e.g., 
contact, impact), especially of those with brittle outer layers [25]. Such structures 
are: TBC for aerospace applications, cutting tools, electronic packaging devices, 
and biomechanical systems — natural ones such as shells and teeth and artificial 
ones such as dental crowns. Brittle out-layers can show cracking before the 
expected lifetime has been achieved; therefore, it is very important to study their 
damage behavior, and spherical indentation is a good tool for it. Moreover, it helps 
the designers of layered structures by providing guidelines for choosing the 
optimal material combinations. 


9.2 Microrobot-based Nanoindentation of Electrically 
Conductive Adhesives 


The use of the microrobot-based nanoindentation testing method is a good 
example of microrobotics technology helping materials research. Nowadays, new 
materials have been developed, and they need to be tested and investigated. The 
new materials have smaller dimensions, reaching the nanometer range (e.g., 
protecting nanocoatings). These materials have more and more applications in the 
microsystems technology. Without microrobot-based nanoindentation, the deter- 
mination of their mechanical properties (hardness and elasticity modulus) is almost 
impossible. The use of microrobots offers more flexibility during the tests, because 
they can solve tasks, which normally imply the use of many devices; moreover, 
their relatively small dimensions allow for the miniaturization of the testing 
devices. Consequently, such microrobot-based nanoindentation testing devices can 
be used inside an SEM for performing in situ nanoindentation tests. In this way, the 
tip of the indenter can be easily and precisely positioned on the surface of such 
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small specimens; moreover, the occurred phenomenon (e.g., cracks, plastic defor- 
mations, pile-up, sink-in, or delamination of the coating) can be observed in situ, 
which consequently can help to better understand their behavior and properties. 


9.2.1 Experiments 


9.2.1.1 Material System 

DIN 16920 defines adhesives as non-metallic materials that join assembly parts by 
the adhesion of surfaces and internal strength [26]. They have no electrical con- 
ductivity. An electrically conductive adhesive (ECA) offers the possibility to join 
electronic components with a substrate and, at the same time, to allow their 
electrical connection. An ECA consists of a binder (matrix), which gives mecha- 
nical strength to the ECA, and a conductive filler providing the electrical conduc- 
tivity [28]. The matrix can be a polymer, a polyamide, or a silicone; the conductive 
filler can be silver, gold, nickel, or copper [27]. They are relatively new materials 
that are receiving more and more interest in the microelectronic industry. ECAs are 
an alternative to the lead-based solder adhesives, which have been intensively used 
for the interconnections of electronic components on printed circuit boards. They 
are environmentally friendly, and they have a better creep resistance and flexibility 
in comparison to the lead-based solder adhesives. However, they also have 
disadvantages. For example, warm and moist environments can lead to a 
degradation of their performance. 

In this work, an epoxy-based ECA silver-filled type PC 3002 [29] has been 
used. It has high electrical and thermal conductivity and high reliability. It is a 
thermoset polymer, which means that it is a cross-linked polymer with a three- 
dimensional molecular network structure [28]. The consequence is that such ECAs 
have a higher resistance to deformation at high temperature. Silver is the most used 
filler for manufacturing ECAs, because in comparison to gold, it is cheaper, and in 
comparison to others, it has a higher conductivity and a better chemical stability. 
Moreover, despite the fact that silver oxidizes (other fillers oxidize also), it shows a 
high conductivity [28]. 

Silver fillers are usually fabricated in a flake shape. The properties of the 
uncured adhesive are [29]: 





e Ag Content 83+1.5% 
e Density 4.4 g/cm’ 
e Viscosity” 22-28 Pa's 
e Processing life” ~32h 
e Placing time” ~16h 
e Storage” 6 months 
* Coverage” ~ 100 cm’/g 
where 
1) At shear rate D = 50 s”, plate-cone system with a cone 2°, temperature 
23°C. 


2) Time at room temperature during which the glue can be processed. 
3) Maximum time between paste application and component placement, at 
up to 60/ R.H., at room temperature. 


4) Storage in the freezer at 40°C. 
5) 25 um printed film thickness. 


The properties of the cured adhesive are [29]: 


Curing (Peak temperature): 


Material Nanotesting 283 


Substrates: Al,O3. lead frames, PI — foil 10’/150°C 
Substrates: PES-Foil 207/120°C 
e Volume resistivity < 0.3 mQ:cm 
e Adhesion (after DIN EN 1465) > 8.5 N/mm 
e Elasticity modulus (after ISO 527-2) ~ 3600 N/mm? 
e Temperature stability” 180°C 
e Glass transition temperature ~ 41°C 
e Weight loss during the curing process at 140°C < 0.8% 
e Weight loss while being at 250°C/1h <0.5% 
e Water absorption < 0.19% 
e Impurities: 
Cl < 20 ppm 
Na’* < 10 ppm 
K* < 10 ppm 
e Thermal conductivity >5 W/mK 
e Shrinkage 4.4% 


where 
1) After 1000 hours at 180°C, the adhesion remains nearly unchanged. 


9.2.1.2 Description of the Experimental Setup 

The microrobot-based nanoindentation testing method has a setup which can be 
used both with an SEM and with an optical microscope. In Figure 9.12a, the 
principal components of the setup are shown: a piezoresistive cantilever is used 
for measuring the indentation force; the piezoresistive cantilever also plays the role 
of an indenter, as the indentation has been performed with the help of the 
cantilever tip; the cantilever is mounted in a special holder which assures the 
necessary electrical connections; the holder with the cantilever is carried by means 
of a linear table driven by a Nanomotor® which is also known as an NMT- 
motorized table; it also plays the role of a sensor measuring the indentation depth. 
The piezoresistive cantilever and the NMT-motorized table are described below 
(Sections 9.2.1.3 and 9.2.1.4). It should be noted that the test is performed in the 
horizontal direction (x-direction). Figure 9.12b indicates how a specimen is held 
and how the cantilever is pressed onto the specimen surface. 
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Figure 9.12. Experimental setup: a. principal components of the setup; b. cantilever pressing 
onto a specimen 
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A control system is required for performing the tests (Figure 9.13). The entire 
set-up is controlled by a computer, which evaluates and processes the measurement 
data and the input signals. This PC is connected to the controller of the NMT- 
motorized table. The PC sends data to the controller and reads the encoder signal 
from the NMT table. The output voltage of the Wheatstone bridge of the 
piezoresistive cantilever is AD-converted and measured by a bridge amplifier. The 
SEM also is equipped with a PC. 


manual positioning 


k USB -E 


a 






4-wire measuring bridge 








input signal bridge amplifier controlling PC 
+ position á A/D converter 
ig LAN 


network controller 





Figure 9.13. Control system 


9.2.1.3 The AFM Cantilever 

Nanoindentation tests require very sensitive force sensors. One of the recent tech- 
nical solutions is to use piezoresistive cantilevers, which are most commonly used 
for performing atomic force microscopy, e.g., for determining surface roughness or 
tribological properties of surfaces (like the friction coefficient) [15]. 

In Figure 9.14 [30], a piezoresistive cantilever and its tip can be seen. This 
cantilever is manufactured on a single crystal silicon substrate. The beam length is 
z500 um and it is equipped with piezoresistors mounted in the form of a 
Wheatstone bridge. If the beam is deflected by applying an external force, a change 
in resistance of the piezoresistors can be measured. The change in resistance, and 
consequently the differential voltage of the Wheatstone bridge, can be converted 
into a force signal by performing a calibration described below (Section 9.2.4.2). 
The beam is also furnished with a tip. The pyramidal tip has a height of ~ 17 um; 
the tip radius is ~10nm; the maximum applicable load is ~20mN; the 
measurement resolution is ~ 1 uN [30]. 
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a b 


Figure 9.14. Piezoresistive cantilever: a. overview; b. AFM tip 


9.2.1.4 Description of the NMT Module 

Nanoindentation tests also require very sensitive displacement sensors and 
positioners. A device that can play both roles is the NMT module [31]. In Figure 
9.15, such a module is shown. In principle, it is a motorized table driven by a very 
precise linear motor. The motor can carry loads up to 2 kg with a maximum stroke 
of up to 70 mm with a resolution of ~ 2 nm. The NMT module is equipped with a 
very precise displacement sensor with a resolution of ~10nm, has small 
dimensions (e.g., module type NMT-20: 50 x 26 x 10mm°), and is vacuum 
compatible [31]. 





Figure 9.15. NMT module 


9.2.1.5. Experimental Procedure 

Before performing nanoindentation tests, calibrations of the piezoresistive AFM 
are necessary: the stiffness calibration and the electrical calibration. They are 
described in Section 9.2.2. Nanoindentation tests were performed on flat specimens 
consisting of the ECA material system described above (Section 9.2.1.1). The ECA 
was applied on SEM stubs. The specimens were variously investigated after a first 
curing at 70°C in the oven for 120 minutes, 150 minutes, 180 minutes, 240 
minutes, 300 minutes, and finally after 325 minutes at the same temperature. The 
nanoindentation tests were performed using the setup described above (Section 
9.2.1.2). The maximum indentation depth was up to 1 um. 
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9.2.2 Calibrations 


For using an AFM cantilever as a force sensor, several calibrations are required. 
First of all, the stiffness of the beam of the cantilever must be determined. For a 
beam with a rectangular cross-section, the relationship between an applied load F 
and its deflection d is: 


F=K-d, (9.23) 


where K is the stiffness of the beam. When the beam is pressed against a specimen, 
it is deflected, and consequently, the differential voltage U of the Wheatstone 
bridge can be measured. The relationship between force and this voltage can be 
written as: 


Fe=c,:U, (9.24) 
where ca is the electrical constant of the AFM cantilever. Thus: 


= reed (9.25) 


(9.26) 


Co 
K 

Co 
C 


So, if the stiffness of the beam K is known, its electrical constant can be extracted, 
and the differential voltage U of the Wheatstone bridge is measured as a function 
of the beam deflection. Consequently, their relationship and the constant c are 
known. Such calibrations are described in the sections below. 


9.2.2.1 Calibration of the Stiffness 

There are many methods for determining the stiffness of an AFM cantilever. A 
first category is the so-called geometric method, where the dimensions of the beam 
are used in equations for calculating the stiffness [32]. Other methods are the so- 
called thermal methods, which use the acquisition of the cantilever thermal 
distribution spectrum, written as a function of the cantilever’s resonance frequency 
[32]. 

In this study, one of the geometric test methods has been used. The dimensions 
of the cantilever beam were measured by means of an SEM. The resonance fre- 
quency was determined with a special setup described in [33]. With these 
quantities, the stiffness of the AFM cantilever can be calculated [32]: 


3 


K=2-w(rl-f.y- = (9.27) 
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where w is the width of the cantilever, / is the length of the cantilever, f. is the 
resonance frequency of the cantilever, p is the density of the cantilever material, 
and £ is the elastic modulus of the material of the cantilever beam. 

An example diagram of a test determining the resonance frequency of the 
cantilever is shown in Figure 9.16. 
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Figure 9.16. Example of a measured resonance frequency as a function of the Wheatstone 
bridge output voltage 


9.2.2.2 Electrical Calibration 

As mentioned above, besides the knowledge of the stiffness of the cantilever, an 
electrical calibration is needed. It allows the translation of the measured 
differential voltage of the Wheatstone bridge to a force (in newtons), by using the 
Equations 9.23 to 9.26. The electrical calibration was carried out by conducting 
measurements of the differential voltage of the Wheatstone bridge as a function of 
the deflection of the cantilever beam. In such tests, the end of the cantilever beam 
is pressed against a hard and smooth surface (e.g., silicon wafer) by using the setup 
shown in Figure 9.12. The output voltage is measured as a function of the 
cantilever beam deflection. This calibration is described in detail in [33]. 


9.2.3 Preliminary Results 


9.2.3.1 Dependency on the Hardness of the ECA on the Curing Time 

The results obtained after nanoindentation tests by using the tip of the AFM on the 
above-mentioned ECA specimens are shown in Figure 9.18. A dependency bet- 
ween the hardness of the ECA and the curing time can be seen. The results show 
an increase of the hardness with the increase of the curing time at constant tem- 
perature. 
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Figure 9.17. Example diagram of an electrical calibration test 
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Figure 9.18. Evolution of the ECA hardness with the curing time 


9.2.4 Discussion 


9.2.4.1 Different Tip Shapes 

A severe problem encountered when using the tip of the piezoresistive AFM canti- 
lever as indenter is its insufficient hardness. After several nanoindentation tests, it 
was worn-out (Figure 9.19). In order to avoid this, a change of the tip of the AFM 
cantilever was proposed. In place of its tip, a ruby sphere with a diameter of 120 
um was glued to the cantilever (Figure 9.20). Ruby has a higher hardness than 
silicon. Moreover, the geometry of the tip is different as the radius of the sphere is 
significantly bigger than the initial radius of the silicon AFM tip. 
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Figure 9.19. Worn-out piezoresistive AFM tip after several nanoindentation tests 
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Figure 9.20. Piezoresistive AFM cantilever with a hard ruby sphere as indenter tip 


First tests with this type of AFM cantilever are very promising. After an optical 
inspection, no wearout has been observed. However, additional calibration tests 
are necessary, and they are still in process at this time. 

Finally, the initial setup for nanoindentation has been modified. The setup now 
uses a Berkovich diamond tip for performing nanoindentation tests and a highly 
sensitive load cell (produced by Honeywell). Moreover, the test is performed in the 
vertical direction (z-direction). The load cell was calibrated by using small weights. 
The setup also requires calibration with reference specimens (fused silica and 
sapphire) with known mechanical properties, in order to calculate both the 
hardness and Young’s modulus of the tested material. At least 5 measurements at 
10 different maximum loads must be performed in order to assure a good statistical 
base for the calibrations. They are also in process at this time. A photo of this setup 
is shown in Figure 9.21. 
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Figure 9.21. Overview of the nanoindentation setup with a highly sensitive load cell and a 
Berkovich indenter 


In this case, the positioning of the specimen under the indenter tip is carried out by 
the NMT module in x-, y-, and z-directions. The indenter tip is then pressed onto 
the surface of the specimen by means of a piezostack actuator with integrated 
strain gages. This means that this actuator can measure the displacement of the 
indenter tip. First tests by using this setup are also very promising. The results 
obtained after nanoindentation tests by using the Berkovich tip, on ECA specimens 
and on ceramic substrates, are shown in Figure 9.22. The ceramic shows a steeper 
loading slope compared to the ECA, meaning that a higher load was necessary for 
achieving the same indentation depth for the test on the ceramic, confirming the 
assumption of higher hardness of ceramics compared to ECA. 
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Figure 9.22. Indentation diagrams of ceramic and ECA specimens with the Berkovich 
diamond indenter 
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9.3. Conclusions 


In Section 9.1, concepts of the instrumented indentation test method have been 
presented. Furthermore, its theoretical analysis with respect to sharp and spherical 
indentation has been explained. The theoretical model of Oliver and Pharr [3] for 
calculating the mechanical properties of bulk materials and coatings, based on the 
results of instrumented tests, was also described. Instrumented indentation has been 
recently developed for testing in the range of very low loads, up to 0.5 N, and of 
small depths, from a few nanometers up to 1 um. This so-called nanoindentation 
has many applications for determining mechanical properties of materials, such 
as ceramics, metals, polymers, and human teeth. 

Section 9.2 presents results of microrobot-based nanoindentation tests on 
specimens consisting of an ECA material after a first curing at 70°C in the oven for 
120 minutes and after longer curing treatments up to 325 minutes at the same 
temperature. The components of the ECA material system and its properties in the 
uncured state are described. ECA are relatively new materials that are more and 
more used in the microelectronics industry. Therefore, it is very important to find 
reliable test methods for characterizing such materials. Material characterization is 
a very important issue when new materials are manufactured. Moreover, the new 
branch of microrobotics can help materials science in its investigations. By means 
of the microrobot-based nanoindentation tests, reasonable values for the hardness 
of an ECA can be obtained, and its evolution depending on curing time at 70°C can 
be monitored. An increase of the hardness with increasing curing time has been 
noticed. 

For performing such tests, a setup that uses the tip of an AFM piezoresistive 
cantilever as indenter has been developed, and the required calibrations have been 
conducted. It was found that, despite the fact that the cantilever as a force sensor is 
very sensitive, the wearout of its tip after only few tests is a big disadvantage. 
Therefore, new solutions for overcoming this problem have been proposed. The 
use of a hard ruby sphere as indenter instead of a silicon tip has been proposed. 
First results are promising, and further calibration tests will be conducted. Finally, 
the setup has been modified. The advantage is now that the tip of the Berkovich 
indenter used is made from diamond. Moreover, for its geometry, an established 
theory for calculating material hardness and Young’s modulus is available in the 
literature [3]. For this setup, it is also necessary to perform further calibration tests 
on reference specimens. The results obtained by using this setup are also 
promising. By comparing the slope of the loading stage of the nanoindentation 
tests on specimens with different hardness, e.g., ceramic and ECA, a difference of 
this slope can be noticed. This permits differentiation of the hardness of different 
materials. 

For improving the results of the nanoindentation tests, the use of very sensitive 
load and displacement sensors plays an important role (Sections 9.2.1.3 and 
9.2.1.4). For further experiments, it is proposed to use a novel force sensor that can 
measure force in all three directions. This sensor has been developed at the 
Technical University of Braunschweig for dimensional metrology [34]. The sensor 
consists of a silicon boss membrane, fabricated by chemical etching and furnished 
with integrated piezoresistors [34]. Because the sensor can simultaneously measure 
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forces in all three directions, scratch tests can also be performed. It is expected 
that, by correlating the results of nanoindentation and scratch tests, the mechanical 
properties of bulk materials and the adhesion of nanocoatings can be determined in 
a better way. 
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10.1 Introduction to EBiD 


Since 1974, when Taniguchi coined the expression nanotechnology [1] as a 
description of manufacturing processes, a lot of different techniques for manu- 
facturing on this small scale have been developed. Until that time, manufacturing 
processes on the micrometer scale used to be the limit. Conventional semiconduc- 
tor-processing technologies are mostly limited by the achievable resolution in 
lithography. However, this resolution depends on the wavelength of light — or, in 
general, on electromagnetic waves. In order to process materials on the nanometer 
scale, it is either necessary to develop a new approach in materials structuring 
(often referred to as the bottom-up approach) or to extend the possibilities of 
common techniques, for example by using electromagnetic waves with 
considerably shorter wavelengths. 

Although it is possible — and not extraordinary anymore using today’s research 
facilities — to structure materials by handling single atoms or molecules with 
scanning tunneling microscopes (STM), this processing technology is not 
economically viable due to its long processing time (approx. 10 ms per atom) [2]. 
Still, the most widely used structuring processes for micro- and nanotechnology are 
based on lithography. Especially with regard to 3D structuring, focused beams 
with high-energy density on the impact spot represent a promising technology. 
Common irradiation technologies fulfilling the demands for high energy on the 
impact spot and of short wavelength are UV-laser, ion beams (e.g., focused ion 
beams — FIB), electron beams, or even X-rays. The basic principles of these 
processing technologies for either the deposition or ablation of materials bear on 
the chemical activation of a precursor on the impact spot, that is where the focused 
beam interacts with the substrate. 
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Electron beam-induced deposition (EBiD) can be regarded as a local chemical 
vapor deposition (CVD) technology inside an electron microscope [3]. CVD is a 
(semiconductor) processing technology, where a vaporized precursor is chemically 
activated by means of thermal energy. This energy is commonly induced by light, 
direct heating, or a plasma source (plasma-enhanced CVD — PECVD). For CVD 
processes, the substrate is heated up inside a chamber filled with the precursor 
vapor at a certain pressure. Due to the thermal energy, a chemical reaction sets in 
on the substrate surface and leads to deposition of the non-volatile reaction 
products. Thus, the substrate surface is coated with these reaction products. In 
semiconductor technology, CVD processes are widely used for deposition of 
uniform thin films over a silicon wafer. 

In contrast, the EBiD process is a very localized CVD process, occurring on an 
area of the substrate defined by the electron beam. Most EBiD setups use an 
electron microscope, because the electron beam can be used for localizing the 
deposition site, triggering the chemical reactions and observation of the deposit. In 
most cases, either scanning electron microscopes (SEM) or scanning transmission 
electron microscopes (STEM) are used for the EBiD process, as their beam can be 
focused on a very small area. Positioning and scanning of the beam allows the 
deposition of dots, lines, areas, and even of three-dimensional figures. In [4], an 
overview of three-dimensional nanostructures is given. 

The EBiD process must be seen as a process defined by the interactions 
between the electron beam, the substrate, and the chemical precursor. In Section 
10.2, a detailed description of electron beam generation and its relevant parameters 
for EBiD processes will be given. 

In and around the spot where the SEM’s electron beam interacts with the 
surface, the chemical reaction is triggered and a deposition is formed. In the early 
years of EBiD, the primary electron beam of the SEM was seen as the trigger for 
the chemical reaction. More recent results show that there are other types of 
triggers, such as the secondary electrons generated through the primary beam [5], 
which might have more influence than the primary electrons. The basic equation 
for the layer growth rate of a deposition described in [6] is still seen as a sufficient 
model to describe the reaction. In Section 10.2.3 this model will be explained and 
the role of the secondary electrons will be considered. The possible heating effects 
of the primary electron beam on the deposition will also be discussed. 

A simplified setup for EBiD in an SEM consists of the electron column, pro- 
viding the electron beam, and the vacuum chamber, where the substrate is 
positioned. The precursor for the chemical reaction is provided by a gas injection 
system (GIS). There, the precursor is evaporated inside a reservoir with a small 
vapor outlet. Different types of gas injection systems and evaporation stages are 
explained in Section 10.3. The gas injection system influences the chemical 
reaction significantly, as parameters like precursor flux density determine the 
growth rate to a considerable extent. Different methods of controlling the precursor 
flux are described in Section 10.4. 

In Section 10.5, control concepts for the EBiD process will be presented. 
EBiD can serve as a technology for bonding objects on the nanometer scale and 
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thus be a very important task in automated nanomanufacturing in the SEM. Basic 
control approaches will be discussed, enabling the user to control the geometry of 
the deposits and the process itself. 

From the beginning of EBiD, a lot of research has been done regarding the 
application of EBiD deposits for lithographic masks or for the manipulation of 
electronic circuits. Even so, very little knowledge has been gathered for the 
application of EBiD as a manufacturing technique for mechanical elements on the 
nanometer scale, that is, with respect to the deposit’s mechanical properties. 
Known values from the literature and own measurements will be presented in 
Section 10.6. 


10.1.1 History of EBiD 


The history of EBiD is strongly linked to the history of electron microscopy. As 
soon as specimens have been imaged using electron microscopes, thin conta- 
mination layers on the specimens’ surface could be observed. This contamination 
effect is still a problem in today’s electron microscopy, even though its extent 
could be reduced. The formation of contamination dots, lines, or layers on a 
specimen surface is due to the presence of small amounts of vapor in the vacuum 
chamber. The first researcher known to describe the effect of specimen contamina- 
tion under electron impact in vacuum was R.L. Stewart in [7]. In his experiments, 
he used an electron gun fixed in a vacuum tube with a pressure of about 5 mmHg 
(1.33-10° mbar). He pointed the electron beam, of approximately 200 V, on 
different target materials, to find them contaminated with a thin film where the 
beam hit the target. He proved that these films were carbon compounds and 
assumed that the precursor for these deposits was organic vapor in the vacuum 
chamber. Furthermore, he observed self-deflection effects of the electron beam 
after deposition which he blamed on the charging of the deposit. He went on to 
formulate the first description of electron-beam-induced contamination deposition: 
The electron beam triggers the dissociation of organic vapor where the beam hits 
the target. 

In 1953, a systematic investigation of organic vapor sources [8] with regard to 
contamination experienced in scanning electron microscopy was performed. The 
researchers measured the thickness of the deposited carbon layer on different metal 
surfaces for different sources of organic vapor, such as diffusion pump oil, 
vacuum grease, and different types of gaskets. Additionally, they discovered two 
interesting effects regarding thermal treatment. Firstly, heating up the target leads 
to a significant reduction in contamination deposition. Secondly, using a cold trap 
close to the specimen also leads to a significant drop in deposition rate. 

In [9], possible applications of EBiD are introduced. As in the previous 
publications, the precursor used was of organic composition (silicone pump oil). Its 
major contribution was the proposing of a phenomenological theory about the 
growth rate of the deposited thin films. Based on the equations in [9], it has been 
shown that the growth rate of the deposit depends on the electron current density, 
which is one of the major beam parameters. The second parameter determining the 
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growth rate is the presence of the precursor on the substrate’s surface, which is 
mainly determined by the evaporation stage, a function of the gas injection system. 
In accordance with [8], it has been proven that the substrate’s temperature 
influences the growth rate. However, the theory was based on the electron density; 
and did not distinguish between primary and secondary electrons, which are struck 
out of the substrate by the primaries. In [10], this theory has been refined, taking 
into account the secondary electrons. 

In 1976, Broers [11] described in the formation of 8 nm metal structures, using 
EBiD in a high-resolution SEM. He suggested the use of his specialized deposition 
technique on 10 nm gold-palladium films supported by 10 nm carbon foils for 
mask application in microfabrication processes. He achieved these very thin 
contamination structures due to the reduction of electron scattering effects in 
these very thin substrates. The protecting film on the metal was formed by 
conventional contamination lithography. 

Since the 1980s, H. W. P. Koops has contributed many pieces of research on 
EBiD. In [6], pattern generation in the submicron range has been investigated, 
using metallo-organic substances — ruthenium carbonyl (Ru3(CO)).) and osmium 
carbonyl (Os3;(CO)\.) — as precursors. A very interesting contribution was the 
determination of the gas flux from the gas injection system to the substrate. At 
that time, the main application for EBiD was seen as the generation of lithography 
masks [6, 12]. In the following publications, further possible applications have 
been found; AFM supertips [13], three-dimensional lithography [14], field-emitter 
devices [15], and even magnetic-flux sensors [16]. 


10.1.2 Applications of EBiD 


As has been shown in the previous section, EBiD started as a troublesome effect in 
SEM imaging. However, over the years, it developed into a highly sophisticated 
processing technology on the nanometer scale. Typical applications are described 
in the following paragraphs. 

Lithography: As already mentioned, [9] proposed EBiD as a coating 
technology for producing insulating and dielectric thin films on substrates. With 
the first experiments using metallo-organic precursors it was possible to deposit 
conducting structures. For example, [17] proposed to use EBiD for the generation 
of etch masks, for mask repair, and also for direct rewiring on IC-prototypes for 
test applications. Due to the high theoretical resolution of the electron beam, 
especially when compared to common lithography tools, it offers a wide range of 
possibilities, at least for non-high-throughput systems. [18] proposed an online 
nanolithography system using EBiD technology in an e-beam lithography system. 
However, the minimum feature size was as high, some 25 nm. 

3D structuring and nanomanufacturing: Based upon the early experiments 
on the three-dimensional deposition of nanostructures [14], this field developed 
into one possible application for EBiD. A typical example of structuring is the 
generation of supertips for atomic force microscopy [16, 19], where comparatively 
simple geometries (pin-like deposits) are deposited. However, more and more 
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complex deposits have been generated, showing the possibilities of this technique 
[20], but also generating new devices [21]. The combination of deposits with 
different mechanical properties has the potential to generate new mechanical 
devices. An example for flexible hinges will be given in Section 10.4. Another 
field of application in nanomanufacturing is the use of EBiD as a bonding 
technology for attaching objects like carbon nanotubes to each other. Typical 
examples can be seen in [22] and in [23]. 


10.2 Theory of Deposition Processes in the SEM 


In this section, the aim is to describe the state-of-the-art regarding the theory 
behind electron-beam-induced deposition and the interactions between electron 
beam, substrate, and precursor, which lead to material deposition. For the 
understanding of the deposition process, it is necessary to recognize the effects of 
secondary electron generation. Major electron beam parameters influence the 
deposition significantly, which will be described in the following section, with the 
single effects such as the generation of secondary electrons and their range. Based 
on this, a description of a deposition growing process can be formulated. 
Afterwards, the rate equation of EBiD and its relevant parameters will be 
presented. After discussing the single parameters, the influence of heat during 
deposition will be taken into account. 


10.2.1 Scanning Electron Microscopy for EBiD 


For most applications of EBiD, SEMs are used, although TEMs (transmission 
electron microscopes) and STEMs are also possible instruments. In this chapter, 
the application of EBiD will be restricted to SEMs, but the fundamentals can be 
applied for other types of electron microscopes as well. 


10.2.1.1 Generation of the Electron Beam 

Inside the electron column of an SEM, the electron beam is generated, shaped, and 
deflected. The extraction of electrons out of the cathode can be achieved using 
three physical effects. These are thermionic emission, field emission, and Schottky 
emission. 

In Figure 10.1, a schematic sketch of a thermionic electron gun is shown [24]. 
The cathode, a hairpin-like wire usually made of tungsten, is heated with an 
electric current to a temperature T, of 2500-3000 K [24]. At that temperature, the 
electrons’ kinetic energy is high enough to overcome the work function of 4.5 eV 
for tungsten. The electrons are emitted from the tungsten wire into the vacuum and 
form a spatial charge. Due to the acceleration voltage U, between cathode and 
anode, the electrons are accelerated towards the anode. For common SEMs, the 
acceleration voltage is between several hundred volts and up to 50 kV. 

The Wehnelt cup between cathode and anode bundles the exiting electron cloud 
to an electron beam. The current of the electron beam is designated emission 
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current 7, with a current density J, of approximately 1.8 A/cm? for tungsten 
hair-pin cathodes [25]. Close to the Wehnelt cup, the cathode, the Wehnelt cup, 
and the anode generate the crossover, which is the beam’s smallest cross-section 
between cathode and anode. After [24], the current density at the crossover has a 
Gaussian profile, which can be calculated using the following equation: 


ID= Je, (10.1) 


where 7, ranges from approximately 10 um to 50 um, depending on the electron 
column parameters. 

Within the last couple of years, field emission guns (FEG) for the generation 
of the electron beam have become more and more popular, due to their improved 
resolution for scanning electron microscopy. Compared to thermionic guns, the 
electrons are extracted out of the cathode by a very high electrical field between 
the cathode and the first anode. This electrical field enables the electrons to tunnel 
through the potential wall. The second anode is used for accelerating the electrons. 
The major advantage of FEGs compared to thermionic guns is the higher current 
density J, at a much smaller crossover size. This leads to better resolution and a 
better signal-to-noise ratio. 
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Figure 10.1. Schematic of a thermionic electron gun as widely used in SEMs. The high 
tension between Wehnelt cup and anode leads to the formation of an electron cloud. The 
crossover is referred to as the virtual source. 
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The third effect used for electron extraction is the so-called Schottky effect, 
which can literally be seen as a combination between thermionic emission and field 
emission. At a temperature of about 1800 K [24], the electron’s kinetic energy is 
high enough to overcome the potential wall of 2.7 eV, which is lowered by the 
electric field. 

The current density J, mainly depends on the emission type of the cathode. 
As will be shown in the next section, the minimum spot size of the beam on the 
substrate has an important influence on the spatial resolution of the EBiD deposit 
and depends on the emission current density. 


10.2.1.2 General SEM Setup 

In Figure 10.2, a typical setup of an SEM is shown [26]. The two main parts are the 
vacuum chamber, where the specimen and the electron detectors are positioned, 
and the electron column. The chamber is evacuated to a pressure around 10° mbar 
by a turbo-molecular pump and a rotary pump. The electron column holds the 
necessary parts for the beam generation, consisting of the cathode and anode and 
for the shaping and deflection of the beam (apertures, condensor lenses, and scan 
coils). From the cathode, the primary electrons are released and accelerated by the 
potential difference U, between cathode and anode. The potential difference 
determines the electron’s kinetic energy, which is mostly quoted in keV. 

The generated beam is bundled by the condensor lens, which leads to the so- 
called crossover. This is also called the virtual source, which is then demagnified 
by the condensor lens(es) and the objective lens on the specimen. The objective 
lens focuses the beam on the specimen. The size of the demagnified crossover 
image is the so-called spot sized,, which determines the SEM’s resolution. The 
spray diaphragms prevent a contamination of the lenses and the scan coils. 

The final aperture determines the current J, passed on the specimen and thus 
has to be set in accordance with the resolution. The aperture diameter and the 
working distance WD, which is the distance between the aperture and the 
specimen, determine the aperture anglea : 


a = tan” ey i (10.2) 
2WD 


The aperture angle œ determines the depth of focus and the resolution. A lower 
aperture diameter D leads to a reduced probe current and to a smaller aperture 
angle. The consequence is an increase in resolution and in focus depth. However, 
the aperture angle can only be reduced to the point where the signal-to-noise ratio 
gets unacceptable for imaging. 

The generated beam is scanned by magnetic fields, which are generated by the 
scan coils. A scan generator controls the deflection and scans the beam line by line 
over the specimen’s surface for generating an image of the specimen. A reduced 
scan speed leads to a better signal-to-noise ratio. The smaller the scanned area, the 
higher the magnification, which can be as high as 300,000 times for an SEM. 
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Figure 10.2. General setup of an SEM, showing the course of the electron beam and the 
relation between the aperture angle, the aperture, and the working distance. Further 
information regarding the image processing can be found in Chapters 4 and 5. 


By positioning the electron beam on a spot, it is possible to generate pin-like 
deposits with EBiD, whereas scanning of the beam can generate line or layer 
deposits. 


10.2.1.3 Secondary Electron Detector 

When the high-energy primary electrons (PE) hit the substrate, low-energy 
secondary electrons (SE) are generated; the mechanisms for this will be explained 
in more detail in the next section. However, the number of generated secondary 
electrons strongly depends on the substrate material and geometry. The generated 
SEs are counted by an Everhart-Thornley detector, whose collector accelerates the 
emitted SE towards the detector with a voltage up to 400 V. The high voltage 
between collector and scintillator of about 10 kV accelerates the electrons on the 
scintillator, where a luminescent layer emits photons when being hit by electrons. 
The emitted photons are guided by a light conductor to a photomultiplier, where 
the light signal is amplified and converted into a current conducted through a 
resistor. The voltage drop on the resistor corresponds to the brightness value of the 
spot where the primary electron beam hits the substrate. By correlating the SE 


Nanostructuring and Nanobonding by EBiD 303 


detector signal with the beam position on the substrate by reading the scan 
generator, an image of the specimen’s surface can be constructed. 


10.2.2 Interactions Between Electron Beam and Substrate 


For understanding the deposition mechanisms in EBiD, deeper investigations of 
the reactions between the primary electron beam and the substrate are necessary. 
As will be shown, the primary electron beam parameters have a major influence on 
the deposition of materials. Furthermore, different substrate materials lead to 
different results in growing experiments. 


10.2.2.1 Energy Spectrum of Emerging Electrons 

Due to the high vacuum in the vacuum chamber, the primary electrons hit the 
specimen with approximately the same energy as that with which they have been 
accelerated. Depending on the substrate’s material parameters such as composition, 
density, and geometry, the penetration depth of the PEs vary. When penetrating the 
substrate, the electrons are scattered elastically and inelastically, which leads to a 
characteristic energy distribution within the substrate and on its surface. During 
this scattering process of the PEs in the substrate, the PEs lose their energy. The 
lost energy is used for the generation of electrons, electromagnetic radiation, and 
heat. In Figure 10.3 [24], an overview of the emitted radiation is given and the 
electrons with their characteristic energies are described. 

Backscattered and low-loss electrons: Due to elastic and inelastic scattering 
of the primary electrons on the substrate atoms, the electrons lose part of their 
energy and change their direction until their energy is lost completely or until they 
escape at the substrate’s surface. A distinction is made regarding the electrons’ 
energy when they leave the substrate. Electrons with energies higher than 50 eV 
are referred to as backscattered electrons, whereas low-loss electrons are defined as 
electrons with an energy close to that of the primary electrons. In Figure 10.4, the 
energy distribution of electrons emerging from the substrate surface is given. 

Auger electrons and X-rays: When inelastic scattering of a primary electron 
leads to ionization on an inner electron shell of the substrate atom, this shell is 
filled up by an electron from one of the outer shells. During this process, energy is 
released in the form of X-rays and so-called Auger electrons. The energy of the X- 
rays and of the Auger electrons is very characteristic of the substrate material and 
can be measured using specialized detectors (energy dispersive X-ray analysis = 
EDX-analysis or Auger electron detectors). The energy spectrum of the Auger 
electrons is between 50 eV and 2 keV. 

Secondary electrons: Inelastic scattering of the primary electrons can lead to 
the emission of electrons from the substrate material itself. These emitted electrons 
are mainly SE that are generated when PEs strike electrons out of the outer shell of 
substrate atoms by lifting up the shell electrons with an energy level high enough 
to overcome the Fermi level. Depending on the energy of the primary electron, it 
can trigger many ionization effects. 
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Figure 10.3. Interaction between primary electrons (PE) and substrate, which leads to 
electron radiation (secondary electrons (SE), backscattered electrons (BSE), Auger electrons 
(AE), and X-rays (X)) 


Based on their origin, secondary electrons only have very limited energy (less 
than 50 eV) and can only leave the substrate when they are generated close to the 
surface. Secondary electrons and backscattered electrons are differentiated with 
respect to their energy. About 70% of secondary electrons only have an energy of 
less than 15 eV [24, 25] (Figure 10.4). 

Specimen current and charging effects: Due to the electron beam impacting 
on the substrate, different electrons are generated in the substrate, as explained 
above. However, not all primary electrons leave the specimen. These are adsorbed 
and dissipated to the ground through the specimen holder. The current flowing 
between specimen and ground, known as the specimen current, can be measured 
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Figure 10.4. Energy distribution of emerging electrons 
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using a pico-amperemeter. However, charging effects of the substrate may occur if 
the resistance between substrate and ground is too high. This can either be due to a 
low-conductivity specimen or to a bad electrical connection between specimen and 
ground. If fewer electrons leave the specimen — as SEs, BSEs, or specimen current 
— than are being injected by the primary beam, charging also occurs. In this case, 
optimizing the beam parameters can reduce charging effects. 


10.2.2.2 Range of Secondary Electrons 

In the early years of EBiD, it seemed to be common sense to consider the electrons 
of the primary electron beam as responsible for triggering the chemical reaction of 
the precursor. However, within the last decade, several observations indicated that 
the reaction is triggered by the secondary electrons. In 1993, [27] noticed that the 
diameter of a pin-like deposit, which is fabricated in spot mode, is about one to two 
magnitudes bigger than the diameter of the PE beam. [28] identify the high 
influence of the secondary electrons on the deposition process with a higher cross- 
section of the secondaries (energies below 50eV) compared to the primary 
electrons (25 keV). [5] confirm this by simulating the growth process of EBiD 
deposits with the Monte-Carlo method. The simulation results are in good 
agreement with their experiments. Based on these considerations, further 
investigation regarding the secondary electrons and their different generation 
effects, intensity, and energy spectrum, is necessary. 

SE1 and SE2: Secondary electrons are defined as low-energy electrons with an 
energy below 50 eV. Especially with regard to EBiD, not only is their energy of 
considerable interest, but also their spatial distribution when emitted from the 
substrate. The generation of SEs is attributed to the inelastic scattering of the high- 
energy electrons. Secondary electrons generated due to inelastic scattering of the 
primary electrons on the substrate atoms are called secondary electrons | (SE1). 

However, due to the elastic scattering of primary electrons (backscattered 
electrons and low-loss electrons), these electrons can diffuse several um away from 
the impact spot of the primary beam (Figure 10.3). During their travel, these high- 
energy BSE can generate secondary electrons as well, which are then called SE2. 
In Figure 10.5, an overview over the different generation spots for secondary 
electrons is given. 

Furthermore, the backscattered electrons can again leave the substrate and 
generate secondary electrons (SE3) far away from the first impact (e.g., on the 
vacuum chamber). However, this effect will be neglected here, as it does not 
contribute to the deposition on the substrate. 
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Figure 10.5. Generation of SE] and SE2 [29]: The primary electrons (PE) hit the substrate 
generating backscattered (BSE) and secondary electrons (SE1). The backscattered electrons 
can again generate secondaries (SE2). 


Spatial distribution of the SE1: Secondary electrons are limited by their low 
energy, and they can, therefore, only leave the substrate when they are generated 
within a depth 2 of the surface [24, 29]. A formal description of the most probable 
escape depth can be found in [30]: 





A =2.67 A (10.3) 


29 


PZ’ 


Where / is denoted in A. A, is the atomic weight, J the first ionization energy in 
eV, p the density denoted in gcm? and Z the atomic number. Based on this 
formula, escape depths ranging from approximately 0.5 nm for cerium to 10 nm for 
carbon can be calculated. Reimer reports [24], escape depths between 1 and 30 nm 
for metal oxides and alkali halides. Based on this, Seiler delivered in 1983 an 
approximation for the spot diameter of the primary excited secondary electrons 
[29]: 


Ase) = Vd +4, (10.4) 


where d, is the diameter of the primary beam and 4 is the mean escape depth of 
the SE1. The diameter of the focused primary beam d, on the substrate can be 
calculated using an equation from [31]: 
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4-kg'To-I 
d, = ee, (10.5) 
m-a- Jg'e Uo 


where kẹ is the Boltzmann constant, T, is the cathode temperature, Ip is the 
beam current, œ the aperture angle, J, the current density, e the elementary 
charge and U, the acceleration voltage. 

Spatial distribution of the SE2: The spatial distribution of the emitted SE2 is 
determined by the range of the backscattered electrons. The range of the BSE is 
furthermore limited by the amount of energy lost per impact, the primary energy 
and the density of the material. The dependence of the relative range R, on the 
primary energy can according to [24] be formulated as: 





R, =aE", (10.6) 


where a and n are material- and energy-dependent parameters which can again be 
taken from [24], eg, a=11.5 and n=1.35. However, [24] notes that the 
dependence on materials is low and is thus neglected for calculations here. Æ 
denotes the energy of the primary electrons in keV. The result R, is denoted in 
ugem’. The absolute range R can be formulated as the fraction of the relative 
range R, and the material density p in gcm” [24]: 


pe. (10.7) 
p 


Based on this equation, the maximum range of a backscattered electron can be 
estimated and consequently the maximum distance from the impact spot of the 
primary beam where SE2 can be generated is defined as #. When the distance 
between backscattered electrons and the primary beam increases, its energy 
decreases and thus its ability to ionize substrate atoms. Seiler established in [29] 
that the spatial distribution of the SE2 released by BSE can be approximated by 
Gaussian distributions. Based upon measurements from [32], the full width at half 
maximum (FWHM) intensity of the SE2 yield can be estimated to approximately 
3 um on silicon at 20 keV primary beam energy. This value rises to 5.8 um at 
30 keV. 

Secondary electron yield: According to [24], the relationship between the 
number of primary electrons N,, and the total number of secondary electrons can 


be defined through the yield factor 6 : 
Ng, =N. (10.8) 


The yield factor 6 is dependent on the energy of the primary electrons Ep; and a 
factor 6 which can be taken from tables (e.g., [30, 33]). The reason for the 


max ? 


decreasing SE-yield with increasing energy, that is, acceleration voltage, is the 
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increasing penetration depth of the primary electrons, which handicaps the deeper- 
generated secondaries from reaching the surface. 

Seiler worked from published experimental data in [29] and derived an 
equation, with which the secondary electron yield ô can be approximated: 


5 =0.86-5' (Epp). (10.9) 


The energy of the primary electrons is denoted in keV. The equation above can be 
used for primary electron energies E,, greater than Epgma:: 1000 eV. Especially 
with regard to the formation of the deposit, the intensity distribution of the 
secondary electrons is of major interest, because it determines the spatial resolution 
of the deposition. In Figure 10.6 [29], a typical intensity distribution of the 
secondary electrons is shown. It is noticeable that the emission distance of the SE2 
can be much wider compared to the emission distance of SE1. In order to find out 
about the correlation between the spatial resolution of the deposition and the spatial 
distribution of the secondary electrons, a comparison of the secondary electron 
yield density between the SE1 and SE2 is necessary. 

The secondary electron yield’s mean density for an electron beam of 20 kV 
acceleration voltage, and a beam diameter of 130 nm must therefore be compared 
for silicon and gold as substrate material with respect to the contributing SE1 and 
SE2. The overall secondary electron yield 6 can be calculated by summing up the 
contribution of the SE1, reflected by6,,, and the contribution of the SE2 6,., 
multiplied by the backscattering coefficient 7 [29]: 


Ô = Ong + NO psp > (10.10) 
where 6,,, can be calculated from the following equation [29]: 


Oxse = Pops - (10.11) 
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Figure 10.6. Schematic intensity distribution of the secondary electrons (SE1 and SE2) 
against the distance from the point where the primary beam hits the substrate 
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And, according to [24], 6,, is proportional to: 
Opp © Epp sec(p)-, (10.12) 


Where ø is the incident angle of the electron beam and the area surface normal of 
the substrate, and / is the exit depth of the SE. The backscattering coefficient 77 
is a material-dependent parameter. Values can, for example, be found in [29] and 
[24]. From [29], the value taken for silicon is 0.22 , and for gold 0.45 . According 
to the calculations of [29], the ratio 8 , which describes the number of secondary 
electrons released per backscattered electron, can be assumed as 2 for electron 
energies higher than 5 kV. Using these values, the average SEI yield density can 
be calculated for silicon as 1.47 x 10° m?, and for gold as 1.77 x 10° m”. 
Accordingly, the average SE2 yield density for silicon is 1.48 x 10'°m” and for 
gold 1.01 x 10'?m”. It is obvious that the SE1 yield densities for silicon and gold 
are in the same region. One reason for this is the low influence of the escape depth 
A (assumed 30 nm for both materials) compared to the high beam diameter of 
130 nm, if typical values for thermionic guns are assumed. However, the average 
SE2 yield density of silicon is about two orders of magnitude smaller than for gold. 
The main reason for this is the reduced range R of the backscattered electrons in 
gold (approx. 0.34 um) compared to that of silicon (approx. 2.8 um). From these 
simple calculations, the following conclusions can be drawn: 


1. The SE1 yield density is very similar for both substrates and therefore the 
influence of the substrate material should be low for beams generated with 
thermionic guns. 

2. The influence of the substrate material on the SE2 yield density is quite 
strong. The result is the effect of a socket formation for depositions on 
high-density materials like gold, as it can be seen in Figure 10.19, where 
pin-like depositions have been made on silicon substrate coated with about 
40 nm of palladium-gold, which is bigger than the exit depth of the SE. In 
contrast to these sockets being formed while depositing on high-density 
materials, thin circular contamination layers several um in diameter form 
when depositing pins on light materials such as silicon, as shown in Figure 
10.7. 


10.2.2.3 Results 

Based on the assumption that secondary electrons play the major role in the 
deposition process, the relevant generation processes for SE] and SE2 have been 
explained. The lower limit for the lateral dimension of an EBiD deposit can thus be 
approximated by the diameter of the primary electron beam, corrected by the range 
of the SEI (/). Depending on the major beam parameters, which are user- 
adjustable on a common SEM, i.e., the acceleration voltage U,, the probe current 
Ip, and the working distance WD, this limit can be calculated using Equations 
10.3-10.5. However, the range enhancement of the SE1 due to scattering of the 
primary beam is very small compared to the beam diameter of thermionic SEMs. 
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This additional material parameter A only comes into account in high-resolution 
imaging using field emission guns or a substrate with low density, where the 
parameter A cannot be neglected against the beam diameter d,. In our 
experiments, the probe current at constant acceleration voltage and the acceleration 
voltage at constant probe current, respectively, have been varied. For acceleration 
voltages above 10 kV and probe currents below 10 nA, the difference between 
beam and pin diameter is approximately between 200 and 400 nm. 

The generation of SE2 due to the inelastic scattering of backscattered electrons 
certainly is a major drawback for high-resolution depositions, as these SE2 can be 
generated up to several um away from the spot where the primary beam hits the 
substrate. This effect can only be minimized when the substrate is very thin, e.g., a 
foil [5]. This results in a very low backscattering coefficient. However, if bulk 
substrates are used, the spatial intensity of the SE2 is very low compared to the 
SE1. This can lead to the deposition of a socket, which can be seen in Figure 10.19. 


10.2.3 Modeling the EBiD Process 


In Figure 10.7, typical EBiD deposits are shown. These pin-like deposits are 
generated when the SEM’s electron beam is pointed on a spot and allow 
comparative experiments for the process parameters. 


10.2.3.1 Rate Equation Model 

In 1986, [6] developed a rate equation model, which describes the growth of 
planar layers by EBiD, depending on precursor parameters and electron beam 
parameters. This phenomenological model has, until now, been the most widely 
used theoretical approach for describing this growth. In Figure 10.8 [34], the EBiD 
process is sketched. The precursor with flux density F,,, in molecules /(s-m’) 
strikes the surface and is adsorbed with a probability given by the sticking 
coefficient s. The adsorbed precursor molecules stay on the substrate surface with 
a mean stay time of tr in seconds. The molecule density in one monolayer is 
designated by the parameter N, in molecules-m~. The cross-section for 
dissociation under electron bombardment is given by o in m’. N,,, denotes the 
density of adsorbed precursor molecules on the substrate surface in 
molecules: m” . Finally, the influence of the electrons is expressed by the electron 


flux density J, in electrons /(s-m”) . 





Figure 10.7. Example of EBiD deposits on a silicon wafer from a tungsten-hexacarbonyl 
precursor. These “pins” have been deposited by pointing the SEM’s electron beam on one 
spot for several minutes. 
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Figure 10.8. Schematic illustrating the EBiD process, showing the different interactions 
between precursor molecules and substrate, secondary electrons and precursor and the 
deposit. a. The gaseous precursor is adsorbed on the substrate surface. b. The primary 
electron beam interacts with the substrate and leads to the emission of secondary electrons, 
which dissociate the adsorbed precursor molecules into volatile and non-volatile fragments. 
c. The non-volatile fragments form the deposit. 


Based upon these parameters, [6] proposed the following equation for the 
change in adsorbed precursor molecules: 


N 
dN p,e = F (1-2 | Na ig J A (10.13) 


On the right side of the equation, the first term describes the density of adsorbed 
precursor molecules, the second term the density of desorbed precursor molecules, 
and the third term the density of dissociated molecules, i.e., the density of the 
deposited molecules. 

Assuming a steady state in Equation 10.13, i.e., dN,,,/dt=0, the density of 
the adsorbed precursor molecules in the equilibrium state N,,, can be calculated 
[12]: 





None = No (#*)/ (v). (0.14) 
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This equation reveals that the maximum surface coverage is one monolayer [12], 
unless the precursor pressure in the gas phase is very high (condensation) [35]. 
Furthermore, Scheuer gives a description of the growth rate Rp, of the 
deposited layer, which is proportional to the volume V, of the deposited molecule 
T m’, the dissociation cross-section o , the adsorbed precursor molecule density 
Npe and the electron flux density J, [6]: 


=V, o Npe J,. 


e 


(10.15) 


Roe 


Substituting N,,, in the equation above with the steady-state precursor density 
Npe from Equation 10.14, the growth rate can be calculated: 





=V, s Foye | (1+ Be). (10.16) 


Row 


Based upon this rate equation model, two cases similar to chemical vapor 
deposition processes can be distinguished: 


e 6tsF,,, < N,: in this case, the surface is never covered with a monolayer. 
The chewitcal reaction happening in the EBiD process is diffusion-limited, 
analogous to low-pressure CVD processes (LPCVD). Thus, increasing the 
precursor flux should result in a significant increase in the growth rate. 

e tsF,,,2N,: the surface is always covered with a precursor monolayer. 
The finiting factor is the electron flux. A further increase in precursor flux 
leads to only a small increase in the deposition rate. This regime is 
comparable with atmospheric pressure CVD (APCVD), where the process 
is energy-limited. 


The influence of the molecular flux on the deposition rate is discussed in Section 
10.4 and can be seen from the experiments shown in Figure 10.14 and Figure 
10.16. These diagrams indicate that the EBiD process using a heated gas supply in 
the high-vacuum chamber shows characteristics of a diffusion-limited process. 
Thus, an increase in the deposition rate can be achieved by increasing the 
molecular flux. However, this is mostly limited by the maximum pressure the SEM 
can tolerate in its chamber. Another approach is the use of environmental electron 
beam-induced Deposition (EEBD), as proposed by [36]. 


10.2.3.2 Parameter Determination for the Rate Equation Model 

The main benefit of the rate equation model is the fact that the deposition rate can 
be calculated from a set of parameters. The main parameters, which can be 
influenced in a simple way, are the precursor flux density F,,,, which will be 
discussed in Section 10.4, and the electron flux density J,, discussed in the 
following section. However, only a little information is given in the literature about 
many of the parameters in the rate equation. One of the reasons for this is the 
challenge of measuring them, e.g., for the dissociation cross-section o for 
metallo-organic precursors under electron impact. 
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Dissociation cross section o: For this parameter, values from 
1.2+0.2=10°°m at 30keV [17] to 0.2+0.15x 107m’ at 40 keV [6] for 
tungsten-hexacarbonyl (W(CO),) are found in the literature. The dissociation 
cross-sections given were measured originally with respect to the energy of the 
primary electron beam. However, the dissociation cross sections peak at low 
energy (below 100 eV), which has been determined by fitting EBiD growth 
measurements. Other values in the literature are mostly found for hydro-carbon 
complexes, e.g., 4.8 x 10°? m? at 25 eV for CH5 [34]. 

Sticking coefficient s: The sticking coefficient s is often found in the 
literature set to one, due to a lack of data. This coefficient is dependent on the 
substrate surface coverage ©, i.e. [35]: 








s=s, (©). (10.17) 


Langmuir proposed a simple model, where the sticking coefficient drops linearly 
with the surface coverage. A further approach assumes that the sticking coefficient 
is constant until a surface coverage of 0.3 to 0.4 and then drops linearly [35]. 
Under this assumption, dissociation is not yet implemented, which might play a 
significant role in the EBiD process, especially as the sticking coefficient cannot be 
modeled without temperature dependence. 

Mean stay time r: The mean stay time describes the desorption rate of 
precursor molecules from the substrate surface. This parameter strictly depends on 
temperature, i.e. [35]: 


Edes 


B= ter (10.18) 





where T, is the pre-exponential factor, E „is the desorption energy, R is the 
molar gas constant, and 7, the substrate temperature. The mean stay time is 
usually considered to be 10°'°s [37, 35], whereas measured values for the pre- 
exponential factor 7, range from 10°'°s to 10°’ s [35]. [6] determined the mean stay 
time during the EBiD process for Ru3(CO),. and Os3(CO),. by fitting measured 
values to 1.4 1 s. 

Deposited molecule volume V„: The volume of a deposited molecule 
depends on the morphology of the deposit, i.e., the crystal structure of the deposit. 
Once the density and the atomic composition of the deposit is known, the volume 
can be calculated easily. Density measurements have been carried out in [38]. 

Molecule density in a monolayer N, : The molecule density in a monolayer 
can be estimated based on the assumption that molecules on a surface are packed 
as densely as possible. If the molecule is considered to be spherical with a radius 
r , the molecule density can be calculated as follows [35]: 





1 


N, = ‘ 
$ 23r? 





(10.19) 
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Applying this equation to the tungsten-hexacarbonyl precursor with a molecule 
diameter of 7.2 10'°m, the monolayer density is 2.23 x 10'*m*. A value 
determined from EBiD measurements is 7.4 x 10" m”? [6]. 


10.2.3.3 Influence of the SE 
Based on the assumption that the secondary electrons are responsible for 
triggering the reaction, the electron flux density J, should be substituted by the 
secondary electron flux density /,, . For pin-like deposits, the deposition rate Rp, 
should then be proportional to secondary electron flux density. 

By contrast, the width of the pins should be equivalent to the minimum 
resolution. The secondary electron flux density J,, can then expressed as: 





: (10.20) 


where 6 is the secondary electron yield, /,is the probe current and d, is the 
diameter of the e-beam on the substrate (spot size). 

Taking into account the fact that the total SE-yield 6 is within the spot size 
dominated by the contribution of the SE1 (Figure 10.6), the SE-yield 6 can be 
substituted by the SEl-yield 6,,. However, there is no simple method for 
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Figure 10.9. Height of pin-like deposits from different precursors and on different substrates 
vs. the relative flux densities of the SE1. The relative SE-flux density has been increased by 
increasing the acceleration voltage of the primary beam. Although this leads to a decrease in 
SE-yield (Equation 10.12), the reduction in spot size (Equation 10.5) is more influential on 
the SE-flux density. All other parameters, e.g., beam current, apertures, working distance, 
and those of the GIS, have been kept constant. 


Nanostructuring and Nanobonding by EBiD 315 


determining the SE1l-yield, and so it is assumed that the secondary electron flux 
density is related to the secondary electron flux density of the single experiment at 
acceleration voltages of 20 kV. The deposited height and thus the vertical growth 
rate shows linear behavior for an increasing secondary electron flux density in 
Figure 10.9. This indicates that the growth rate is also energy-limited. 


10.2.3.4 Heat Transfer Calculations 

In the previous section, the fundamental equation for describing the deposition 
process in EBiD has been shown. Many of the parameters are approximated or 
calculated from deposition experiments. One of the main parameters determining 
the deposition rate, which is strongly dependent on temperature, is the mean stay 
time r. This parameter describes the average time a molecule rests on the 
substrate surface until it is either desorbed into the gas phase or disassociated. The 
mean stay time strongly depends on temperature, as shown in Equation 10.18. A 
raised temperature of the substrate will lead to a reduction in the deposition rate. 

The influence of heat generation on a sample under observation in the SEM, 
and the possible damage through heating, have been estimated in [24]. Based upon 
his calculations, the effects of heat generation on the deposition are restricted to the 
case when the generated heat cannot dissipate due to a reduction in the heat 
dissipating cross-section or an increase in the insulating length. This is, for 
example, the case for pin-like deposits. 

Assuming a pin-like deposit of diameter D , length L and heat conductivity 2 , 
the temperature difference AT between the cold end of the substrate and the hot 
end irradiated by the electron beam can be calculated using the following equation 
[24]: 


_ 4P,L 


AT =— 75, 
TAD 


(10.21) 


where P, is the power dissipated into heat in the deposit. The heating power P, 
can be estimated from the electron beam power P,, by a factor f between 40 and 
80% for plain substrates according to [24]. Simulations based on the Monte- 
Carlo method showed heat dissipation factors f approximately between 10 and 
20% for pin-like structures, due to the scattering of electrons through the pin. 
Within these simulations, the number and energies of electrons entering a pin-like 
structure have been balanced against the number and energies of electrons leaving 
the structure again. 

Simulations from [39] showed that temperature increases of 30 K for pin-like 
deposits from a TEOS (tetra-ethoxy-silane) precursor are possible for certain 
structures. Experimental data for deposits of other precursors is not available, 
mostly due to the missing data on the thermal conductivity 2 of the deposited 
material. 
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10.3 Gas Injection Systems 


In this section, an overview of gas injection systems will be given. These systems 
allow the user to inject a gaseous precursor stream into the SEM’s vacuum 
chamber, out of which the deposition on the substrate’s surface will be made. 
Depending on the application, different gas injection systems will be discussed. In 
order to determine the mass flow and its flow characteristics, a model will be 
described, explaining the different flow characteristics and their influence on 
deposition parameters. The need for a mobile GIS will, therefore, be discussed. 
Additionally, techniques and methods for pressure control will be described. As an 
application showing the possibilities of these systems, combined depositions are 
presented. 


10.3.1 Introduction 


Gas injection systems deliver the evaporated precursor to the deposition spot. Gas 
injection systems can be distinguished as being of mainly two types — internal 
systems situated inside the vacuum chamber and external systems which are 
flanged on the outside of the vacuum chamber. 

Requirements for gas injection systems: For the deposition, one major 
parameter is the density of the precursor flux j,,,. This parameter determines the 
amount of precursor molecules per solid angle and time unit (molecules -(sr-s)”' ) 
provided for the chemical reaction. As will be shown in the following sections, this 
is one of the most crucial process parameters. An increased molecular flux of the 
precursor also leads to a rise in the system vacuum, and so the mass flow cannot be 
allowed to rise above a certain level determined by the vacuum system. In order to 
allow high precursor density on the spot at low increase in vacuum chamber 
pressure, the molecular beam should be focused or “peaked”. As the flux decreases 
with increasing distance between the capillary outlet and the deposition spot, 
another attribute is the ability to minimize the distance. However, this distance is 
also dependent on the geometry of the capillary and cannot be reduced to zero for 
most applications. Especially for handling and assembly applications, where 
bonding with EBiD is one process in a chain, it is necessary to switch off or to 
reduce the precursor flux in order to prevent contamination while imaging with the 
electron beam. 

Precursors: In the experiments presented here, only metallo-organic precursors 
like tungsten-hexacarbonyl (W(CO).) or di-cobalt-octacarbonyl (Co(CO)s) have 
been used. These metallo-organic precursors are sublimates, i.e., at room 
temperature and ambient pressure they are available as a solid powder, although 
they have to be kept under a protective gas atmosphere in order to prevent 
dissociation. These sublimates evaporate directly from the solid state, and the 
evaporation rate can be controlled by temperature for a given pressure. The 
metallo-organic precursors can easily be dissociated by the EBiD process, which 
leads to depositions composed of metal, carbon, and oxygen. 
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10.3.2 The Molecular Beam 


10.3.2.1 Modeling of the Mass Flow Between Reservoir and Substrate 

In Figure 10.10, a schematic drawing of an evaporation system for metallo-organic 
precursors is shown. This evaporation system is situated in the SEM’s vacuum 
chamber. The precursor fills the reservoir and evaporates through a capillary of 
length / and diameter d into the vacuum chamber. The precursor flow leaving the 
capillary outlet forms a molecular beam pointed towards the deposition spot on the 
substrate. 

The characteristics of the precursor flow through the capillary can be described 
using the Knudsen number Kn , which is the ratio between the molecular mean 
free path 4 of the precursor and the characteristic dimension of the capillary — the 
diameter d : 


A 
Kn=—. 10.22 
ts (10.22) 


Depending on the Knudsen number, three flow regimes can be distinguished [37]: 


e For Kn>0.5, the flow is of molecular type, and the mean free path of the 
molecules is very long compared to the capillary’s geometry. Thus, the 
interactions between the single molecules are very low and collisions of the 
molecules on the capillary’s surface affect the flow. This flow regime is 
present, therefore, in the SEM’s (high) vacuum chamber and in the outlet 
of the capillary. 

e The other extreme is a very low Knudsen number, i.e., Kn < 0.01. In this 
regime, the interactions between molecules dominate the flow. This can be 
the case for high pressures in the capillary inlet. 

e Between those two extremes, the so-called transitional regime is situated 
(i.e., 0.01< Kn < 0.5). This is the case in the capillary before the molecular 
flux at its outlet is reached. 


The mean free path 2 can be calculated using an equation formulated by 
Maxwell [35]: 


1 


E 
V27o°n 


(10.23) 


where o is the diameter of the molecule and n is the molecule density. The 
mainspring for the precursor flux between the reservoir and the vacuum chamber is 
the difference in pressure, i.e., the reservoir pressure p,,. and the pressure inside 
the vacuum chamber p „. The pressure along the capillary, starting from the inlet 
and ending at the outlet, decreases from p,,, to P, continuously. Thus, the mean 
free path of the molecules changes as well as the flow regimes. However, for 
different flow regimes, the conductance of the capillary also changes. In the 
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Figure 10.10. Schematic of an evaporation system for metallo-organic precursors 


following paragraphs, these conductances are determined in order to calculate the 
molecular flux from the reservoir to the vacuum. 

Figure 10.10 shows how the flow regime can change from the capillary inlet to 
the outlet. The sketched regimes are strongly dependent on the pressures along the 
capillary, and also at the borders, when flow regime changes cannot be strictly 
defined. For the calculation of the mass flow, it is reasonable to start the 
calculation with the outlet part of the capillary, because its conductivity is the 
dominant factor on the mass flow. 

Molecular flow on the capillary outlet: In 1961, Becker [40] published his 
results on the formation of molecular beams with channel sources, which emit into 
vacuum. Based upon the theory in [41], the effective length /,, of a long circular 
tube, which defines the distance between the capillary outlet and the section where 
the collision probability of a molecule is 50% , has been introduced. The effective 
length can be calculated using the following equation, which considers the 


blocking of the molecular flow at the capillary outlet due to the formation of a 
precursor cloud [40]: 


nee i TER =R (10.24) 
3 64 N pe O 
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where d is the capillary diameter, v is the mean gas velocity, N,, is the 
molecular flux and o is the molecule diameter. Based on this, the molecule 


density 7, at the capillary section at position /—/,,, can be calculated [40]: 





24N preg | 16N Pre 

N eis lag + da (10.25) 
From the equation above, the pressure in the capillary at the position where the 
flow changes from the transitional to the molecular regime p,_ Ip Can be calculated 
by using the ideal gas law for ideal gases. 

Transitional flow: Between the capillary inlet and the capillary section in a 
distance /,, from the outlet, the flow can be considered transitional. For this flow 
regime, the pV -throughput can be calculated from [35]: 





m 1 d Pi- Pia, m- Ë 
~ 128° y; l l 2 P l "(Pres Pizy )> (10.26) 
oe off ~ eff 


Av 


where Z is a dimensionless correction factor, first established by Knudsen [35], 
and c is the mean molecule velocity. The q, -throughput can be transformed into 
the molecular flux Xp, for ideal gases using the ideal gas law. 

Flux calculation algorithm: Based upon this calculation scheme, the 
interesting parameters molecular flux Np, and effective length /,, can be 
calculated using an iterative algorithm. In the first iteration (i =1), the molecular 
flux N 5. between capillary inlet and outlet is calculated using the inlet pressure 
P,e, and the outlet pressure p,,., which can be assumed to be 0 (capillary 
exhausts into vacuum). Then the effective length a and the pressure Pii at 
capillary position /—/,, are calculated. Using these values, in the second and 7 -th 
iteration step, the molecular flux M7” in the transitional regime is calculated for 
a capillary of length /—/',"", inlet pressure p,,,, and outlet pressure Pit, ". The 
iteration steps are repeated, until the change in the values becomes negligible. 

The above scheme is for capillaries where the Knudsen number at the inlet is in 
the transitional regime, which has been the case for the GIS and precursors 
described above. Thus, possible viscous flow, which might be the case for high 
inlet pressures or very small capillaries, is neglected. However, the algorithm can 
easily be extended for this case. 

Directivity of the molecular beam: In the previous section, a method for 
calculating the molecular flux jy,,, between the precursor reservoir and the 
deposition spot has been described. In general, the deposition rate is proportional to 
the quantity of precursor molecules on the deposition spot and thus to the 
molecular flux. However, the maximum molecular flux is limited by multiple 
parameters. Generally speaking, the molecular flux should be kept as low as 
possible, in order to reduce contamination of the SEM, namely the vacuum system 
and the electron gun. These devices can be damaged by precursor reaction and 
contamination. In order to achieve high precursor density on the deposition spot at 
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a minimum molecular flux, it is necessary to peak the molecular beam, generating 
a beam with very high directivity and thus a high molecular flux density jp,- 

Lambert emitter: The molecular flux density or intensity j,,, relates to the 
solid angle emitted from an aperture of diameter d and thickness /, where the 
thickness is very small compared to the diameter. This is referred to as the Lambert 
emitter. The flux density is strongly dependent on the angle @ measured 
perpendicularly from the aperture and can be described using the following 
equation [41]: 


Jpre(9) = COS(A) jpe (0 = 0°). (10.27) 


The Lambert emitter is, due to its cosine dependence, also known as the cosine 
emitter. The unit of /,,.(@) is molecules-(sr-s)'. The molecular flux density at 
angle 0 =0° can be calculated from the molecular flux X p, [40]: 


Jimme (0 = 0°) = Ne (10.28) 
T 


Peaking factor: In order to compare a molecular beam source regarding its 
directivity, a comparative factor y is defined, which is the ratio of the peaked 
center line intensity jee, (0=0°) to the intensity of a cosine emitter 
Jere „ (9 =0°) with the same leak rate [42]: 


a) 
Tog. (0 z 0°) l 


y= J Pre (10.29) 


After [40], the maximum possible directivity 7, of a capillary with a diameter d 
and length / can be calculated using the following equation: 


6l 
=1+—. 10.30 
Xo 37 ( ) 


If the capillary length / is substituted with the effective length /,, from Equation 
10.24, the peaking factor y for a given gaseous precursor flowing through a 
capillary with dimension and d with a molecular flux j/;,, can be calculated. 
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Figure 10.11. SEM image showing the substrate being partly coated with a circular 
deposition. The precursor was evaporated through the capillary looming from the lower 
image edge to the substrate edge. The circular deposition spots show the cosine-dependence 
of the molecular flux evaporating through the capillary outlet. 
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Figure 10.12. Calculated peaking factor for tungsten-hexacarbonyl and di-cobalt- 
octacarbonyl precursors vs. molecular fluxes for a capillary of 40 mm length and different 
inner diameters. Reasonable peaking factors can only be achieved by a reduction of 
molecular flux. 


In Figure 10.12, the calculated peaking factor y vs. the molecular flux for 
different precursors and capillary diameters is shown. The calculations demonstrate 
that high peaking factors can only be achieved if the molecular flux is reduced, or 
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that for a given molecular flux, the diameter has to be widened. The main reason 
for this is the blocking of the capillary outlet, as concluded in [40]. Consequently, 
a compromise between molecular flux and peaking factor has to be found. 


10.4 Mobile GIS 


The precursor flux density on the deposition spot Jp, i.e., at distance r from the 


capillary outlet, is strongly dependent on the distance r: 





Ine (1) = we (10.31) 


where the distance r is denoted in m and the precursor flux density J,,(r) at 
distance r is denoted in molecules-(s-m*)'. Hence, the distance between the 
capillary outlet and the deposition spot has a major effect on the precursor flux 
density and thus on the deposition rate. A system for adjusting this distance and 
controlling the precursor flux will be presented in the following sections. 


10.4.1 General Setup 


In Figure 10.13 the setup for the EBiD is shown. Inside the vacuum chamber, 
one or multiple gas injection systems are positioned. 
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Figure 10.13. Schematic of the EBiD setup. In the SEM’s vacuum chamber, the substrate 
and a mobile GIS are situated. The GIS can be positioned with the mobile platform and its 
controller. Closed-loop pressure control is achieved by using the pressure sensor for 
feedback, heating and cooling, respectively, of the GIS. The deposition on the substrate can 
be controlled by the beam controller, i.e., positioning the beam and reading the SE-detector 
signal. 
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These systems are positioned on mobile platforms and so the capillary outlets 
can be positioned close to, and oriented to, the deposition spot. The pressure in the 
vacuum chamber can be measured by a separated full-range pressure gage, which 
is flanged to the vacuum chamber door. The substrate is positioned, rotated, and 
tilted using the SEM’s stage in all three dimensions for observation with the 
electron beam. In principle, the gas injection systems are positioned so that they do 
not conceal the SE-detector. 


10.4.2 Position Control of the GIS 


A major parameter for the deposition is the distance between the capillary outlet 
and the deposition spot. In order to allow high flexibility, the GIS has to be 
transportable, which ensures that this distance can be adjusted with respect to the 
deposition task. As has been shown previously, the molecular flux density 
decreases with increasing distance between capillary outlet and deposition spot. In 
this way, the influence of the distance on the deposition rate has been measured. In 
Figure 10.14, the dependence of deposition height on distance between capillary 
outlet and deposition spot is shown. During this experiment, all other parameters 
have been kept constant. 

As can be seen from Figure 10.14, the dependence between deposition height 
and distance from the GIS outlet is very strong. Thus, it is necessary to minimize 
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Figure 10.14. Dependence of the height of the deposited pin-like structures from the 
distance between capillary outlet and deposition spot for tungsten-hexacarbonyl. All pin-like 


structures have been deposited with the same process parameters. Only the distance has been 
varied. 
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the distance between the deposition spot and the capillary outlet. In most cases, 
this can only be achieved by using a mobile gas injection system, which is 
adjustable in three axes. Furthermore, by withdrawing the capillary from the 
deposition spot, the precursor flux on the deposition spot can quickly be reduced to 
zero, which is an effective method for minimizing contamination. 


10.4.3 Pressure Control 


According to Equation 10.31, the precursor flux density at distance r from the 
capillary outlet depends on both, the distance r and the precursor flux density jp,- 
The latter can be calculated using the flow calculation algorithm described in 
Section 10.3.2. From Equation 10.26, it is obvious that the pressure difference 
between the precursor reservoir and the vacuum chamber is the driving force for 
the molecular flux. In this section, different types of evaporation systems will be 
presented and compared. Based on this, methods for controlling the precursor flux 
will be shown. 


10.4.3.1 Constant Evaporation Systems 

One of the simplest setups for evaporating a metallo-organic precursor in the 
vacuum chamber of the SEM is a small precursor container, which emits a 
precursor flux through a small capillary on the substrate. If the container is 
thermically well connected to a heat reservoir (e.g., the vacuum chamber walls), it 
can be assumed that the container’s temperature does not change significantly 
during the evaporation of the precursor. Thus, the pressure in the container only 
depends on the vapor pressure of the precursor at the given temperature of the 
container and on the conductance of the capillary. 

This setup is easy to realize and may serve for deposition experiments where 
process parameters are evaluated. In order to interrupt the precursor flux to the 
substrate, the reservoir and capillary can be moved away from the substrate using a 
mobile platform or similar actuating device. However, the precursor will still be 
evaporated and contaminate the vacuum chamber. Apart from this disadvantage, 
the molecular flux from constant evaporation stages for a given precursor is 
determined by the geometry of the capillary. The molecular flux density on the 
deposition spot can only be controlled by the distance between capillary outlet and 
deposition spot. 

The diameter of the capillary has to be chosen small enough to allow the 
vacuum system of the SEM to reach the system’s operating pressure, but wide 
enough to allow sufficient precursor flux. If the diameter chosen is too wide, the 
vacuum pumps will evacuate the reservoir to a considerable extent, until operating 
pressure is reached. 


10.4.3.2 Heating/Cooling Stages 

In order to overcome the above disadvantages of constant evaporation stages, gas 
injection systems, using Peltier elements for heating and cooling the precursor 
reservoir, have been developed. In Figure 10.15, a gas injection system for metallo- 
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organic precursors is shown. The precursor fills the reservoir, which is sealed by a 
rubber o-ring between reservoir and cap. A steel capillary with length 40 mm and 
an inner diameter of 0.6 mm is connected to the reservoir. Heating and cooling, 
respectively, of the reservoir can be achieved by shifting heat between the thermal 
mass and the reservoir. In order to control the temperature of the reservoir, a 
temperature sensor has been built between the reservoir and the Peltier element. 
This sensor also prevents the overheating of the precursor during the heating-up 
process. A second temperature sensor has been built into the column carrying the 
capillary, for measuring the temperature difference between the reservoir and the 
capillary. In order to prevent possible condensation effects in the capillary, it is 
supported by a column, which ensures a low temperature gradient along the 
capillary and reservoir even during the heating-up process. 


10.4.3.3 Control of the Molecular Flux 

The precursor is evaporated in the reservoir of the gas injection system by applying 
a positive voltage on the Peltier element, which heats up the reservoir and cools 
down the thermal mass. For fast disruption of the molecular flux from the reservoir 
to the deposition spot, the polarity of the Peltier element’s driving voltage is 
switched, and the reservoir is cooled. This allows fast temperature cycles and 
prevents precursor leakage through the capillary if no deposition is wanted. For 
controlling the molecular flux from the reservoir to the deposition spot, the gas 
injection system can be operated in two modes, either by closed-loop control of 
the reservoir temperature or by closed-loop control of the vacuum chamber 
pressure. These two modes will be explained in the following paragraphs. 
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Figure 10.15. Gas injection system for metallo-organic precursors. The precursor is 
evaporated through the capillary (1) pointing to the deposition spot. The reservoir (2) holds 
the precursor and is heated and cooled, respectively, by the Peltier element (3). It shifts the 
heat between reservoir and thermal mass (4); the temperature can be controlled by a sensor 
(5). The GIS hole can be positioned and oriented towards the substrate using the mobile 
platform (6). 
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Temperature control: In temperature control mode, the temperature sensor 
between the Peltier element and the precursor reservoir is used as closed-loop 
sensor input. If the heating-up process to a distinct temperature is finished and the 
evaporation system can be considered isothermal, the molecular flux and thus the 
pressure in the vacuum chamber and on the deposition, can be considered constant. 
However, experiments showed that a constant pressure is very hard to achieve 
using temperature control. There are a couple of reasons for this effect. As the 
vapor pressure of metallo-organic compounds is strongly dependent on tempe- 
rature (vapor pressure can be approximated by an exponential law), a small change 
in temperature can lead to a dramatic change in molecular flux and thus vacuum 
pressure. These small changes in temperature can also be based on the tolerances 
of the temperature sensors. Furthermore, change in pressure can be caused by the 
inertia of the control system itself, as the heat is slowly distributed in the 
evaporation stage compared to the velocity of the evaporation process. 

Pressure control: During the experiments with temperature control, it became 
obvious that a pressure control system is better suited for EBiD. For sensor 
feedback in the control loop, a full-range pressure gage has been flanged to the 
SEM’s door, which enables the measuring of the pressure in the vacuum chamber. 
Furthermore, the SEM’s vacuum gage values have been gathered, in order to reveal 
pressure differences. However, the pressure gage is gas-dependent and thus the 
measured values can only be correlated to the true pressure by means of a 
correction factor. Although the pressure sensor is far away from the deposition 
spot, compared to the distance between capillary outlet and deposition spot, it can 
be assumed that a high pressure correlates to a high precursor flux. 

For controlling the pressure in the vacuum chamber, the pressure gage value is 
used as sensor in the feedback loop. Based on the difference between actual 
pressure and pressure set point, the voltage was adjusted on the Peltier element 
using a PID controller. 


10.4.3.4 Pressure Dependence of the Deposition Rate 

Changing the vacuum chamber pressure by introducing the precursor is a 
parameter that is very simple to use for closed-loop control of the evaporation 
stages. However, the measured pressure can only be evaluated as a relative value, 
because the pressure sensor is calibrated against nitrogen. Unfortunately, 
correction values are not available for the precursors used. 

A pressure increase is caused by an increase in precursor flux, which leads to a 
higher adsorption rate of precursor molecules on the substrate surface. This again 
leads — according to the rate equation model — to a higher deposition rate, as long 
as the EBiD process is in the diffusion-limited region. However, the possible 
pressure range is limited by two boundaries: the lower boundary is the minimum 
pressure the vacuum system of the SEM is able to generate; the upper limit is the 
maximum pressure before the vacuum switches off the SEM for safety reasons, 
when, for example, there is overheating of the turbo-molecular pump or danger of 
short circuits in the cathode of the e-beam system. 
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Figure 10.16 shows measured values for heights of pin-like deposits. The 
experiments were carried out on a plain silicon wafer as substrate, under the same 
conditions and parameters, except for pressure. All deposits clearly indicate that 
with rising precursor flux and thus pressure in the vacuum chamber, the deposition 
rate rises as well. 


10.4.4 Multimaterial Depositions 


Nanostructuring with EBiD is very flexible with respect to the geometric 
dimension. Pin-like deposits can be generated by pointing the electron beam on 
one spot on the substrate, line deposits by scanning the beam in a line, and layer 
deposits by scanning an area. Out of these basic geometric elements, more complex 
systems can be built up. By combining deposits with different physical or chemical 
properties, sensory and actuator elements can be built. In the following list, 
examples for possible applications are given: 


e Flexible hinges: combination of three pin-like deposits on top of each 
other, where the one in the middle is made of a more flexible material. 

e Thermal actuators: two pin-like deposits made of materials with different 
thermal elongation coefficients, situated beside each other and linked by a 
bridge on top of both, where one of the pins is deposited on a circuit path 
for heating. 
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Figure 10.16. Height of pin-like deposits made from tungsten-hexacarbonyl and di-cobalt- 
octacarbonyl precursor at varying vacuum chamber pressures 
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e Thermocouples: two line-deposits with two materials crossing each other 
in one point. 

e Strain gages: laminar deposit of a non-conducting material, on top of 
which lines of a metallic material are deposited. 

e Sensory elements: two laminar deposits on top of each other. 


Three-dimensional deposits can also be realized by scanning the electron beam 
slowly while depositing. For generating sensor or actuator elements, it is 
advantageous to combine deposits of different precursor materials. Therefore, it is 
necessary to introduce multiple precursors alternating on the deposition spot. This 
can be achieved by exchanging the precursor, but this is not very flexible. For this 
reason, a setup has been developed that allows almost seamless change in the 
precursor flux. Two independent gas injection systems are positioned around the 
substrate, one containing tungsten-hexacarbonyl, the other di-cobalt-octacarbonyl. 
An EDX-microanalysis of the deposited structure can be seen in Figure 10.17. 
This combined deposition forms a flexible hinge. In bending experiments, it has 
been found that the middle part is much more flexible than the bottom and top 
depositions. 

The gas injection system provides the precursor flux, which is the most 
important and influential part in the EBiD setup of the primary beam. Based upon 
theoretical considerations regarding precursor flux, its density and directivity, it 
has been shown that mobile evaporation stages that are heated and cooled, 








Figure 10.17. EDX-analysis of a pin-like multimaterial deposition of tungsten-hexacarbonyl 
(l and 3) and di-cobalt-octacarbony] (2). The deposited pin has a length of approx. 4 um. 
This combination forms a flexible hinge, as cobalt deposits are much more flexible than 
tungsten deposits. 
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respectively, are helpful for optimizing the deposition process with respect to the 
growth rate and low contamination of the vacuum chamber. 

One of the most promising technologies for increasing the precursor flux 
density seems to be the application of miniaturized multichannel arrays, which 
promise a better directivity. 


10.5 Process Monitoring and Control 


In recent years, EBiD has developed into a promising nanostructuring technique. 
Various applications have been reported, ranging from rapid prototyping on the 
nanometer scale to nanolithography. Many examples have been published, 
showing the application of EBiD for three-dimensional structuring. In [27], the 
manufacturing of supertips on top of an AFM tip has been shown. This application 
has been extended by many others, e.g., the manufacturing of high-resolution 
magnetic supertips [19] and submicron hall devices [43]. Further examples for 
three-dimensional structuring are given in [20]. In the field of nanolithography, 
EBiD has several advantages compared to FIB techniques, because no damage is 
caused in the observed substrate [17]. This fact means that EBiD is especially 
suitable for the repair of lithography masks, which was proposed by Koops et al. as 
early as 1987 [12]. Mask repair technology based upon EBiD developed into full- 
scale wafer technology [44], which can be integrated into today’s semiconductor- 
processing technologies. 

These more and more complex applications of the EBiD process demand 
sophisticated methods of process control. In this section, methods for controlling 
the EBiD process with respect to automation will be given. 


10.5.1 Time-based Control (Open-loop Control) 


The most simple method for controlling the deposition process is a time-based 
technique. In Figure 10.18, the deposited tip height vs. the deposition time is 
shown. From this diagram, the growth process can be divided into two parts. At 
first, the deposition starts with a very high rate which drops continuously to a 
constant growth rate in the second part. The slope of the deposition rate vs. time in 
the first and in the second part is strongly dependent on process parameters such as 
the precursor, its flux density, and electron flux density. However, guaranteeing 
constant process parameters enables the user to control the deposition rate. 

For two- and three-dimensional deposits, respectively (e.g., deposits generated 
through line scans and area scans), the residence time ¢, on a spot can be 
calculated from the scan speed between electron beam and substrate v,,, and the 


diameter d,,, of pins deposited with equal process parameters: 


dy, 
tp =, (10.32) 
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Figure 10.18. Tip height vs. deposition time. The single pins have been deposited using the 
same process parameters, and only the deposition time has been changed. 


Based upon this equation and a constant growth rate, the deposited height can be 
approximated. However, this method only allows open-loop control of the 
deposition and strongly depends on the knowledge and accuracy of the process 
parameters for accurate depositions. 


10.5.2 Closed-loop Control of EBiD Deposits 


In order to overcome the disadvantages of time-based control, especially for 
deposition heights which are in the non-linear growth period (Figure 10.18), a 
closed-loop control method has been developed. This method comprises the 
evaluation of the SE-detector signal, which is a standard detector for any SEM. In 
[45] the correlation between probe current, SE-signal, and deposition height for 
pin-like structures is described. 

Based on the time variable-change of the detector signal during 1-D deposits 
and its correlation with the deposition height, a method has been developed to 
control the deposition height during 2D depositions. Furthermore, the evaluation of 
the detector signal enables the user to track down certain process failures during 
the process. 
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10.5.2.1 Growth of Pin-like Deposits and SE-signal 

Pin-like deposits are grown with EBiD, when the SEM’s electron beam is 
positioned on one spot of the substrate’s surface and a gaseous precursor is 
provided. In Figure 10.19, the results from typical growing experiments are shown. 

The single deposits with a deposition time of up to 5 minutes have a Gaussian- 
like shape. After approx. 5 minutes, the tip starts growing out of the surface and 
the deposit has the shape of a needle. However, the tip geometry is still the same. 
The growing process can be separated into two distinctive parts. In the first interval 
(in Figure 10.18 from 0 to 5 min), horizontal and vertical growth takes place, until 
the tips’ Gaussian-like shape is completely developed [45]. In this interval, the 
height growth rate is non-linear. It starts with a high growth rate, which decreases, 
until a constant rate is reached. The second interval is characterized by linear 
growth. The time when the non-linear growth changes to linear growth depends on 
several process parameters such as substrate material, beam parameters, and 
precursor flux density, when the tip profile is formed. 

In Section 10.2, a description of the interactions between the specimen and the 
electron beam has been given. The quantity of secondary electrons generated on a 
sample depends, among other parameters, on its geometry. During the growing 
process of an EBiD deposit, the geometry changes, as can be seen in Figure 10.19. 
This leads to a change in the SE-detector signal and in the probe current signal. 
However, the signal change is only measurable until the deposit’s tip shape is 
formed and only until vertical growth of the pin happens [45]. This correlation 
between the SE-detector signals can be used as an in situ process control method 
during the deposition. The SE-signal S (x,t) for a pin-like deposit at time ¢ ata 
position x on the substrate can be determined using the following function: 





Sigs A-SE Date :13 Apr 2006 A AAD 
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Figure 10.19. SEM image of deposition experiments using tungsten-hexacarbonyl as 
precursor on silicon substrate coated with palladium-gold. The rows show different 
deposition times (3, 2, 1, 5, and 10 minutes from left to right). Within a row, the deposition 
time was equal. 
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Sor (x,t) = Sug (X) + Spe (t) > (10.33) 


where S g(x) is the signal from the substrate during observation at position x . 
This can, for example, be achieved through imaging the deposition spot. However, 
the imaging time should be small compared to the deposition time. The signals are 
strongly dependent on the detector parameters, e.g., brightness and contrast, but 
also on beam parameters. Sp,(t) is the time-dependent signal, while the pin is 
growing. This signal can be determined using an exponential law [45]: 


Sop © = So (=e) ; (10.34) 


where S, is the maximum signal achievable while growing the pin for certain 
detector parameters and T stands for process-dependent parameters. They reflect, 
for example, the substrate material, the precursor parameters like chemical 
composition and flux density, as well as the beam parameters. The growth function 
for a pin-like deposit in the non-linear growth regime can then be denoted by: 


h(t) =L-S,,,(0), (10.35) 


where L is a process-dependent constant, which has to be evaluated by parameter 
variation. 


10.5.2.2 Application for 2D Deposits 

Based upon the previously described relationship between the SE-detector signal 
and the growth of one-dimensional (i.e., pin-like) deposits, a method for closed- 
loop control of two-dimensional (i.e., line) deposits has been developed. If the 
electron beam is scanned along a certain trajectory X on the substrate, and the SE- 
signal along this trajectory is recorded, the function S,,(X) is established. In the 
simplest case, this can be achieved by using the "line scan" functionality of the 
SEM control. For a defined process parameter set, the correlation between the SE- 
signal S,,,(¢) and the deposited height A(t) is known from deposition 
experiments. The residence time ¢, can be calculated from Equation 10.32 and 
thus be controlled by the scan speed Ven using a control algorithm. 

In Figure 10.20, an SEM image of line deposits using the algorithm described 
above is shown. Three lines have been deposited as a bundle with the same height. 
On the right edge of the image deposited pins can be seen, which have been used 
for recording the growing function A(t) and the corresponding SE-signal function 
Sp.»(t). From these functions, the relationship between sensor signal and 
deposited height S(/) can be calculated. 
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Figure 10.20. SEM image of deposited lines of different heights using the height control 
algorithm 
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Figure 10.21. Signals during line deposits. Points on the line marked with a cross represent 
the mean value of the detector signal during the line prescan. Points marked with a circle on 
the line represent the mean value of the detector signal during deposition with a scan speed 
given by the reciprocal of the x-axes. The thick line marks the mean height of the deposited 
line with respect to the right y-axes. The dashed line represents the difference in the detector 
signal between prescan and deposition, which shows the proportionality between detector 
signal during deposition and line height. 


The relationship between scan speed, SE-signal, and deposited line height is 
shown in Figure 10.21. The signals along each deposited line and the height of the 
line have been averaged. The prescan signals have been gathered by imaging the 
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line. The distinct relationship between deposition speed and the difference in the 
detector signal between prescan and deposition is very clear. The prescan can be 
seen as a fast scan (tę << T) along the given trajectory where a line is to be 
deposited. In the simplest case, this can be the recording of an image which 
contains the trajectory. 

Based upon the algorithm described above, many relevant applications become 
realizable in closed-loop mode. One example is the height control during line 
deposition, even if the substrate is not plain or has a strong SE-signal contrast. 
Another typical nanomanufacturing application is the in situ control of EBiD bonds 
during deposition. In this case, two objects are connected to each other by 
depositing a line perpendicular over the flange where both objects are in touch. In 
the SE image, this usually gives a strong contrast to the image. Using the algorithm 
described, constant height of the bonding line can be achieved. 


10.5.3 Failure Detection 


By evaluating the detector signal during deposition with EBiD, distinct process 
failures can be evaluated and the process can be stopped or restarted. Typical 
process failures are: 


e Drift effects during deposition: these occur if the substrate or the 
deposition are charged electrically through the primary beam and thus 
cause an electrostatic deflection of the primary beam. Another possibility is 
drift due to thermal elongation effects (heat transfer due to heated precursor 
gases, heating of the deposition) or mechanical instabilities of the setup. 

e Beam generation problems: due to flashover, short circuits or other 
failures in the beam generation, the primary beam is disturbed during 
deposition. 


Both failure scenarios show distinct characteristics in the SE-detector signal during 
deposition. In the case of drift effects, the signal slowly drifts away from its 
reference, as shown in Figure 10.22a. The inlaid picture shows the effect of drift 
while depositing a pin-like structure. For evaluating the signal characteristics, 
upper and lower limits can be defined, between which the signal is allowed to float 
during deposition. If the signal strays outside these limits, the deposition process 
should be restarted, taking into account the fact that that the beam possibly has to 
be repositioned automatically or manually. 

In the case of beam generation failures, the SE-detector signal shows a distinct 
behavior, as shown in Figure 10.22b. The signal characteristics are formed by 
cracks and peaks. In this case, the deposition process usually has to be stopped for 
maintenance on the beam generation system. 
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Figure 10.22. Typical SE signals during deposition, when a. drift or charging happens and 
when b. the beam generation system is disturbed, e.g,. through short circuits 


Based upon the evaluation of the SE-detector signal, it is possible to control the 
height of one- and two-dimensional deposits, i.e,. pin-like and line deposits. Out of 
two-dimensional objects, more complex three-dimensional objects like layers, 
combined depositions, or highly structured objects can be formed using the same 
algorithm. However, the major process parameters must be accurate for the exact 
control of the deposited geometry. The influence of the single parameters can be 
evaluated using pin-like deposits. As has been shown, the control of the SE-signal 
can be used to detect process failures and thus regulate, restart, or stop the process. 
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10.6 Mechanical Properties of EBiD Deposits 


EBiD is often used to build mechanical devices, to connect objects on the nano- 
meter scale to each other, or for electrical connections. Especially with regard to 
bonding tasks in manufacturing processes inside the SEM, it is necessary to 
observe the mechanical properties of the EBiD deposits. In recent years, the 
mechanical properties of deposits have become more interesting. The main reason 
for this is the technology’s potential use for setting up micro- or nanomechanical 
systems. The values determined in the literature and from own experiments are 
presented below. 

Young’s modulus: In [46], EBiD layers from paraffin were deposited. These 
were tested using nanoindentation in order to determine the Young’s modulus. The 
values where measured as 34.3 +3.4, 46.3+2.3 and 59.5+2.5 GPa for 
acceleration voltages of 3, 12 , and 20 keV, respectively. 

Utke et al. [47] estimated the Young’s modulus for EBiD deposits from 
W(CO),- and Co2(CO)s-precursors in the range between 10 and 100 GPa. Wich et 
al. [48] measured the Young’s modulus of pin-like deposits, which were deflected 
using an AFM-cantilever until fracture of the pin for the W(CO).-precursor, of 
between 21.6 and 50.4 GPa. 

Tensile strength: In [47], the tensile strength for deposits from a W(CO),- 
precursor has been determined to be 2+0.5 GPa and for deposits from a 
Co2(CO)s-precursor to be 1 + 0.5 GPa, using an AFM-cantilever-based method. 

Hardness: The hardness of EBiD layers has been investigated in [46] for 
paraffin precursors and in own unpublished experiments for tungsten-hexacarbonyl 
and di-cobalt-octacarbonyl. In both experiments, Berkovich-shaped indentation- 
tips were used. The hardness values were determined for the paraffin precursor to 
be 3.6+0.3, 4.0+0.2, and 4.4 +2 GPa for acceleration voltages of 3, 12, and 
20 kV, respectively. The values for the W(CO),-precursor were measured to be 
7.6°;¢ GPa at 20 kV acceleration voltage. The measured hardness for EBiD layers 
from a Co(CO)s-precursor was 3.713 GPa at 20 kV acceleration voltage. 


10.7 Conclusions 
10.7.1 Summary 


Based on the working principle of an SEM, the interactions between electron 
beam and substrate have been described. The generated secondary electrons can be 
distinguished in SE1 and SE2; methods for calculating their range and the yield 
have been described for both. Typical EBiD phenomena, such as socket formation 
and rings around pin-like deposits, have been explained, with the different 
secondary electron flux densities for the single species. 

The rate equation model described in [6] has been discussed with regard to 
single equation parameters. In the literature, less knowledge applicable for EBiD is 
known for these parameters, complicating the exact determination of the EBiD 
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growth rate. Based on the assumption that the secondary electrons play a major 
role in EBiD processes [5], the rate equation model has been adapted to the 
secondary electron flux density. The height of pin-like deposits made from 
different precursors and on different substrates with different acceleration voltages, 
and thus secondary electron flux densities, showed a linear behavior when plotted 
against relative SE flux density. 

The GIS that delivers the precursor to the deposition spot is a very important 
part of the EBiD process. The crucial parameter for the EBiD process is the 
precursor flux density, which can be optimized by optimizing the capillary 
geometry. A model was introduced for calculating the precursor flux density and 
the peaking factor of the precursor emission. Based on this model, it has been 
shown that a tradeoff has to be accepted for capillary-type GIS between peaking 
factor and molecular flux. 

For optimizing the achievable results with capillary-type GIS, the distance 
between capillary outlet and deposition spot should be minimized. Further 
improvements with regard to process control can be achieved by closed-loop 
pressure control of precursor evaporation. 

Further process control can be achieved by evaluating the SE-detector signal 
during deposition; this is of special interest for line deposits, as the deposited 
height can be controlled in situ. Bonding applications in the SEM can profit from 
this closed-loop control method. Additionally, process failures can be identified 
early by evaluating the SE-detector signal. 

Finally, the mechanical properties of EBiD deposits found in the literature 
and from own measurements have been presented, emphasizing the importance of 
EBiD for micro- and nanomechanical applications. 


10.7.2 Outlook 


The EBiD process has developed over the last 40 years from a disturbing effect in 
electron microscopy to a promising deposition technology. Today, it offers many 
different applications, such as mask repair, nanomanufacturing, or nano- 
structuring. The main reason for this can be seen in the progressively shrinking 
structural size in the semiconductor industry and in nanotechnology. This 
development process will lead to a wider field of application for EBiD, due to its 
miniaturization potential. Especially with regard to the evolution of 
nanotechnology and the resulting need for analysis methods (eg., CNT 
characterization), EBiD is one of the most promising technologies for setting up a 
bonding system on the nanometer scale in the SEM. 

The process itself is defined by the interaction of the primary electron beam 
with the substrate and the precursor molecules and therefore a wide variety of 
process parameters arises from these interactions. Although a basic 
phenomenological description of the deposition process is known, until recently it 
was not possible to describe the EBiD process precisely and in detail. A major field 
of research today still is the influence of heat on the chemical reaction and the 
chemical and structural composition of (and influence on) the deposits. 
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Additionally, for a wider field of applications in industry, the growth rates are too 
slow; further improvements on the gas injection systems seem to be a promising 
approach. 

Utilizing the improvements of the primary electron beam systems, e.g., cold 
field emission guns, further reductions in the lateral size of deposits below 20 nm 
have already been achieved in SEMs. Further improvements in this field will 
certainly increase the application of EBiD in the semiconductor industry. 
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